Monday, April 30, 2018

Flow-Based Economic Simulation

Information, Discovery, and Non-Omniscient AI

HiI had so much progress last week that I felt it would be unwise to wait until this Friday to share. To that end, I started writing a log last Friday. Sadly for me, happily for you, it was too long to finish, so I finished it today! Hope you enjoyLast time, I discussed economic volatility, and how simplistic models & AI can (and do) cause instability and constant cycles of overshooting, correction, over-correction, and on and on. Since then, I've implemented historical data tracking for market items, which allows AI agents to see and act on data aggregated over various time scales.However, while giving more thought to colony dynamics and their relationship to LOD economy simulation (as mentioned last time, they are really the same problem), I began to see the whole problem in a new light. A day later, the economy was humming along with a stability and robustness never-before-seen in LT's history!The insight is simple: it's easier to balance rates than absolute quantities. If you're watching the food supply of a colony, you'll need to observe it for a while to know whether there's a net surplus or deficit, especially if food is subject to lots of noisy processes happening on lots of different timescales (and almost everything interesting is). If, on the other hand, you could see all of those processes listed out with their 'average rate' (in food/day, for example), all you would need to do is sum those rates and you'd know whether, in the long run, there would be a surplus or deficit.But it gets much better than that. Let's think of the entire economy as a flow network -- for a concrete analogy, a system of water pipes (or a circuit, whichever you prefer). At each 'node' (a colony, a station, ...), we can keep track of the flow of economic quantities. Perhaps colony A -- a densely-populated urban capital -- requires 300 tons of grain/day. So we keep track of a flow of grain, at A it has value -300. Think of this like a 'negative pressure' at A (a drain/sink). 10 different traders, each capable of moving (round-trip) 20 tons of grain/day, decide to haul grain from colony B -- a rural and primarily agricultural establishment -- which has a surplus of +200 tons grain/day, to A. Now we update net flow values: at A, we go from -300 to -100. At B, we go from +200 to 0. Obviously, this is a 'good decision' on the traders' part, because they have significantly reduced the total pressure in the economy, bringing it closer to perfect supply/demand equilibrium. Just like current flows naturally between voltage gradients, just as water flows naturally between pressure gradients, so too do economic quantities flow naturally between supply/demand pressure gradients. This whole analogy borders on common sense. And common sense tends to work well when one can find itThe most obvious concern that we might raise about this technique is that it heavily depends on accurate estimation of the result of various economic activities. If traders compute a flow value of +100 per unit time but are only able to deliver +10 in reality, the economy will settle into a wildly-inefficient equilibrium wherein markets are constantly understocked due to what we might playfully interpret as a 'pervasive overoptimism concerning how much can be delivered in a certain amount of time' on the part of the AI. ([clairvoyance] 'Joke' about Josh being a flow-based AI with this very issue[/clairvoyance]) The solution to this is two-fold: first, use real math to estimate things. The AI in LT is already quite accurate in its ability to estimate job impacts. It will virtually never get things wrong by more than a factor of 2, much less by an order of magnitude. Second, to refine accuracy even further, we can compute a corrective term for the calculated flow value based on market data! We can split the supply/demand terms in our flow calculations and use them as follows: suppose a market has computed flow of 15 supply of death sticks, 10 demand for death sticks (per day). Then we expect, on average, for market data to show about 10 trade volume per day, and about +5 total supply volume. If market data tells us that the average trade volume is 50 sticks/day with 0 change in total supply, we can guess that our flow calculation is probably wrong and that ~50 supply / ~50 demand is a better estimate for death sticks. Also, with that many death sticks trading per day, the folks at this market clearly need to go home and rethink their lives.It's worth noting that, even with 100% accurate AI estimates, calculating flow corrections is still necessary since the player can have a sustained impact on the economy, but does not report this impact to market nodes like the AI does.Anywho, that's a lot of theory talk, but...does it work? You bet! It works really, really well. In my simulations, the flow technique quickly finds optimal equilibria, even in complex systems where the optimal economic structure is quite complicated. What's more, since the AI is always thinking about how to optimize the flow / minimize the 'pressure' of the economy, we actually see some interesting dynamics play out as we change the number of assets operating in a system. Watch!In the following simulation, 1000 ships is simply not enough to saturate the total water demand of 8 colonies (their demand is kept constant for the purposes of this simulation; in the real simulation some of those colonies would die out since the economy can't support them all). Still, the AI applies some fluid-dynamics-like reasoning to try to make sure that the colonies are each 'minimally undersupplied.' The resulting equilibrium is quite nontrivial, with some colonies being supplied exclusively through trade, while those in proximity to ice mining locations are supplied directly:(Note: I have a colony selected, and you can see the market EMAs (exponential moving averages) for water there; notice how the price has settled nicely to the 5-6 credit range and has remained fairly stable through most of history. Just as one would expect, the stability of these flow-based economies is crushingly-superior to my previous methodology!)Notice how the AI totally ignores two entire ice fields, which it has (correctly) determined are essentially wastes of time in this system. Of course, when we apply more factors to the simulation, like diminishing returns for overpopulated fields, piracy, AI personality, and the like, we will see more interesting dynamics.I've also introduced variation in size, speed, and cargo capacity to the simulation. The AI correctly takes things like top speed & cargo capacity into account when computing speculative flow values for activities like mining or trading, so in some cases you can actually see interesting patterns emerge from these considerations. In fact, in this shot, you can see one such pattern! Look at the four trade hubs, and the three trade routes connecting them. The traders are basically all tiny! Look at the miners. On average, the trade ships are smaller than the mining ships. Almost all of the trade ships are the minimum size, whereas we see a large variance in miners. I did not code anything that would directly cause or even suggest this behavior. So why does it happen? Given the various constants of this simulation, the AI has reasoned that certain ship properties are more important for trading, while others are more important for mining. Mining ships must sit idly as they extract water from ice. For them, speed is less important than cargo capacity...at least, that's my spot-analysis of what's going on. Note that this isn't indicative of any objective truth -- given different 'universal constants' in the simulation, I would expect the situation to change entirely. The point, though, is that the AI has taken the specifics of the simulation and figured out how to craft optimal behavior with them. Nice.Now, if we crank up to 2000 ships, the situation changes:The colonies can now be supplied adequately, so there is less pressure on the AI to choose optimal water-supplying jobs. In fact, remember that the goal is to minimize differences in supply/demand (net flow) -- so the AI is going to (again, correctly!) select 'bad' jobs for some ships, because doing so ensures that colonies are not flooded with surpluses! This is the only reason for choosing to mine in the far ice fields, which you can see some ships are now doing. It's actually interesting to note that the AI is not applying 'rational capitalist' behavior here, but rather 'rational collectivist' behavior; some units are performing intentionally-suboptimal work in order that the whole can be optimal. Philosophical arguments aside, this works out well for our purposes of simulating a predominantly-AI-driven economyFinally, in a 10,000-ship simulation, the economy is completely over-saturated:Every possible job is seeing heavy activity. Trade develops along most potential trade routes; mining is in full swing at every location. The analogy to a network of water pipes with way too much water flowing in is apt. We can see the spray of ships here saturating the bursting economy in a very literal way.In summary: economic volatility is gone, equilibrium is here, and the AI is generally much more capable of setting up well-structured economies that take into account all of the nuances of the star system and game constants. Long live flow-based economics \o/ As I finish more game mechanics and implement the corresponding AI jobs for them, we will continue to see a richer and richer spectrum of emergent behaviors and economic configurations.With one problem solved, I moved on to the next and began implementing information and discovery mechanics. I've already written quite a bit, so I won't go as deeply into this work, but I'm certain there will be more to come. At this point, I've implemented the fundamentals: entities can be made 'discoverable,' and if they are, a list of players that know about the entity is tracked. Furthermore, for the first time ever, the high-level AI is respecting this limitation on information, which means that an AI agent must know about a zone before it can begin mining there, must know about a market before it can trade there, must know about a wormhole before it can compute a course that uses said wormhole, etc!This is a very exciting step toward delivering on one of the promises that's near and dear to my heart: exploration as a real, profitable job. With AI players restricted and only able to use known information in their high-level planning, the ability to profit from discoveries becomes a natural (even essential) game mechanic.When discussing information mechanics in LT, I am frequently asked the question: "so that means that if you discover a new wormhole and sell the location to AI, the major trade routes, maybe even the entire economy, could change completely?" and of course my answer is: "You bet"Indeed, we're so deprived of this lovely dynamic behavior in single player games that something as simple as AI responding to new information seems downright magical. Like so many things, it is, in truth, far simpler to implement than people imagineSo let's have some proof of that answer, yes? Here we have a 1500 agent simulation in the same system shown above. This time, however, restricted information is turned on. AI players are initially given information of all colonies, but only one ice field. Obviously, this makes the initial equilibrium very, very different from the one shown above, in which the AI knew about all fields.As expected, we see a completely new structure arise due to the information constraints. Again, the structure consists of some colonies being directly-supplied mining hubs, while others are connected through a trade network.Now, I hit a button to give the AI information of all fields in the system. I have been a very busy explorer, mapping out these 5 other fields, which I now sell all at once to have maximal impact on the economyShortly after the new information has been revealed to the AI, the economy begins to break down and disperse. It is preparing to change shape....And after a while, as expected, it comes back to the equilibrium shown in the previous section, since the two simulations are now operating with the same information:Voila! As desired, the discovery of new information can completely re-shape the economy! There's a lot more to talk about when it comes to this mechanic: what information does the AI have to begin with? How often does it discover new information without player interference? Will there be anything new under the sun by the time I arrive? It's too much to discuss at the moment. But I have given much thought to these questions this week and have solid plans for how most of this is going to work.---Just for fun, let's have one final shot with 100 tiny planets. You could imagine that they're stations instead of planets. Let's just see what the AI can come up with in an unrepresentatively-complex system:AmazingThe AI creates a network not unlike several little hearts pumping blood through arteries, forming a mining 'core' around each ice field, then developing trade routes that fan out to reach the far colonies. All of this behavior is emergent and self-organizing. Why does each colony 'belong' to one and only one 'core'? Why do cores structure themselves like little spanning trees? Why do we sometimes see a far colony supplied by both a trade route to a near colony and direct mining from the core (and why do these 'far' mining operations always seem to be conducted by only the largest ships?) In each case, I'm sure we could spend quite some effort analyzing the situation and uncover why the choice makes sense, which pieces of the simulation have contributed to it being optimal, and so on. For me it is already enough to see this behavior and be happy that we have Real Stuff™ driving the gameCapital expenditure was my final todo item from last time, and, while I've developed some theory that I think will work nicely, I haven't yet implemented it and thus won't talk about it in this log. After all, I've already gone on for quite some time. But I will give you a little teaser and say: flow-based economics makes capital expenditure much more tractable and even affords a formulaic way to compute the best investment -- be it weapon upgrades, a new research project, or the construction of a new station -- at any point in time. It's still a challenging problem, but is much easier with flow information.Given that I've spent a fair bit of time on this log, I'm not certain that I'll be posting as early as this Friday. We'll see; if I have lots of exciting developments then you'll hear from me again this week, otherwise count on next week (EDIT: bah, I need another week...apologies ). In the mean time, I'm also going to be getting a KS update out this week (but it won't really be exciting for those who have followed the logs).Farewell o/~Josh