Monday, December 4, 2017

Major breakthrough with being able to use Lua data & functions from our C subsystems

breakthrough with being able to use Lua data & functions from our C subsystems Used the above to implement a C-side scheduler that schedules Lua logic for execution; significantly faster than previous implementation

Ported planets & atmospheres

Integrated Loom for advanced inspection of LuaJIT traces and assembly

Implemented several new algorithms for procedural geometry library, touched on in Lindsey's latest log

Loads of quality-of-life improvements to how 'game-level' Lua is written

First-pass design of mod format taking into account all existing support systems

We're All Modders: Hammering out LT Mod Format for Building Gameplay

Side-Soliloquy: Why So Much Concern for Moddability?

Spoiler: SHOW April 2013 (!!)



Sometime in 2014, the tides began to change. I began to recognize the need for an external data format in which to place 'arbitrary' constants like those relating to high-level balancing (how much energy is required to negate one unit of damage compared to dealing one unit of damage? How much energy is required to extract one energy unit's worth of raw material from an asteroid? etc..) Naturally, it wasn't long before a similar need for 'arbitrary' code was recognized. The solution at the time was LTSL, and, as we all know, thus began the march toward the dark days, FPLT, and some of the hardest months of my life.



Looking back, it's clear that my solutions, from that point on, were guided by an unspoken requirement: LT had to be moddable, on that point I was no longer willing to yield -- else I would surely have eventually come back to C/C++, learned to write a more modular engine with faster compile times, and found a way to iterate on gameplay code without needing a separate language. It's hard to admit. But it's the truth. Hence, today, we are in a world where great sacrifices in development time and effort have been spent on Limit Theory's touted modding capabilities. Personally, I'm OK with that. And here's why.



When others ask about LT, moddability often comes up. Invariably, I bring up Limit Theory's



What I can do? What I'm really good at doing, as this whole process has taught me, and perhaps, what it is that I love doing above all else: it's probably not game design, it's probably not even procedural nebula algorithms ...no...building things that allow other people to realize creation with ease, simplicity, and power...now that is the kind of thing that can get me out of bed in the morning. And that's exactly what I/we have done. We've built something that can take a high-level, simple description of a new gameplay mechanism, and do everything that needs to be done behind-the-scenes to let that mechanism play nicely with a thousand others, in the presences of tens of thousands of entities stretched accross a boundless universe. We've built something that renders many of your suggestions doable in a few hundred lines of Lua or less. And THAT...I'm proud of that



Everyone knows that Morrowind was one of my favorite games of all time. And it still is. It's still cranking, 15 years after release (in 2002). People still play it, people still buy it, people still love it. All because Bethesda decided they would build their game with a Construction Set rather than pure code, thereby enabling others to add to their original vision (which was already fantastic) via that tool. And the result is that today, some of us are still talking about how stunning Morrowind looks with graphics overhaul 3.0 et al; some of us are still in awe that Tamriel Rebuilt is alive-and-kicking, expanding the vanilla game's playable area by god-only-knows how many square miles.



And this is my dream for Limit Theory: that it ships as the vision you all supported in 2012 + the modding capabilities that have since become a core feature, and that, in 10 years' time, it has become something utterly and totally beyond that which I could have done myself or with our small team in the same amount of time. I really can't wait Many of my technical expositions, and even the less-technical ones, are ridden with talk of moddability and concern that a given system will not hinder the extensibility of LT. I feel that it might be a good time to step back and look at the big-picture of LT development for a moment; to express what modding means to me, what it has meant for Limit Theory's development process, and how it has impacted both to the utmost degree. To open, I'm going to make a daring assertion: I believe that Limit Theory would already have been released by this point had I chosen to stick with my original vision of 'no modding.' Take it as you will, perhaps noting that I do have a patently lackluster track record of timeline judgement (:oops:) Still, I feel that there is quite some truth to the statement. After all, we had the solid beginnings of a game with the release of the LT Prototype all the way back in(!!)Sometime in 2014, the tides began to change. I began to recognize the need for an external data format in which to place 'arbitrary' constants like those relating to high-level balancing (how much energy is required to negate one unit of damage compared to dealing one unit of damage? How much energy is required to extract one energy unit's worth of raw material from an asteroid? etc..) Naturally, it wasn't long before a similar need for 'arbitrary' code was recognized. The solution at the time was LTSL, and, as we all know, thus began the march toward the dark days, FPLT, and some of the hardest months of my life.Looking back, it's clear that my solutions, from that point on, were guided by an unspoken requirement: LT had to be moddable, on that point I was no longer willing to yield -- else I would surely have eventually come back to C/C++, learned to write a more modular engine with faster compile times, and found a way to iterate on gameplay code without needing a separate language. It's hard to admit. But it's the truth. Hence, today, we are in a world where great sacrifices in development time and effort have been spent on Limit Theory's touted modding capabilities. Personally, I'm OK with that. And here's why.When others ask about LT, moddability often comes up. Invariably, I bring up Limit Theory's suggestions sub-forums at some point, typically bragging on it as being "surely on of the best places on the internet to find inspiring, space-game/sci-fi ideas of remarkable breadth and depth, elaborated on by people of singular imagination and intellect." Since the KS closed at the end of 2012, there have been >1,000 topics and >20,000 posts in this subforum alone. It is a remarkable place -- one that I wish I had the time to read front-to-back. But that's just it: I don't. I don't even have enough time to read all of the beautiful, elaborately-thought-out gameplay suggestions generated by you all, much less implement them.What I can do? What I'm really good at doing, as this whole process has taught me, and perhaps, what it is that I love doing above all else: it's probably not game design, it's probably not even procedural nebula algorithms...no...building things that allow other people to realize creation with ease, simplicity, and power...now that is the kind of thing that can get me out of bed in the morning. And that's exactly what I/we have done. We've built something that can take a high-level, simple description of a new gameplay mechanism, and do everything that needs to be done behind-the-scenes to let that mechanism play nicely with a thousand others, in the presences of tens of thousands of entities stretched accross a boundless universe. We've built something that renders many of your suggestions doable in a few hundred lines of Lua or less. And THAT...I'm proud of thatEveryone knows that Morrowind was one of my favorite games of all time. And it still is. It's still cranking, 15 years after release (in 2002). People still play it, people still buy it, people still love it. All because Bethesda decided they would build their game with a Construction Set rather than pure code, thereby enabling others to add to their original vision (which was already fantastic) via that tool. And the result is that today, some of us are still talking about how stunning Morrowind looks with graphics overhaul 3.0 et al; some of us are still in awe that Tamriel Rebuilt is alive-and-kicking, expanding the vanilla game's playable area by god-only-knows how many square miles.And this is my dream for Limit Theory: that it ships as the vision you all supported in 2012 + the modding capabilities that have since become a core feature, and that, in 10 years' time, it has become something utterly and totally beyond that which I could have done myself or with our small team in the same amount of time. I really can't wait

Hey again everyone! Hope you all had a good Thanksgiving (for those to whom it is applicable). I certainly ate my fair shareIt's been a productive few weeks since the last log (minus Thanksgiving break), but last week, in particular, saw two major breakthroughs that have me really happy with the current state of affairsAt this point, we've essentially got all of our core support systems working. By that, I mean that we have either Lua-level or C-level (in some cases a mixture) mechanisms for supporting almost everything gameplay code could want to do: from defining new datatypes that can be used with the utmost efficiency, to adding and/or modifying existing ones, to writing gameplay code that needs to run in response to certain events, or needs to be scheduled to run at certain frequencies, to writing new methods that can be called on the aforementioned datatypes, to defining jobs that batch-process large chunks of entities that match a certain filter, and so on. Where there was once a wild-west of Lua trickery, there is now a solid set of production-ready, battle-tested system. So, what remains? Naught but the orchestration of said systems to compose the symphony of gameplay that is Limit TheorySo, it is time that we once again turn our attention to the format for this orchestration: the Limit Theory Mod Format. I have expressed many times that I wish for our vanilla LT gameplay-level logic to be written in exactly the same manner as any other mod, and I'd like to stand by that -- both for the sake of making it easier on us devs (since we'll have a well-defined, decentralized framework for incrementally adding modular bits to the game), as well as for the sake of ensuring that LT is not just moddable in theory, but is built on the very proof that it's moddable in practice.Thus far, my solution is unfolding as a two-part mod format, where each part may exist zero or more times within any given mod file. One part is data, the other part is code. Data sections use a very simple 'data description language' to perform the aforementioned orchestration of our system. A data section for implementing an inventory might look like this:Hopefully it doesn't take too much brainpower to figure out what's going on here: we've opened up some object scope called 'Ship,' declared that it has an Inventory, and then gone on to define Inventory itself. Inside Inventory, we can see three types of 'declarations,' each of which corresponds concretely to a programming concept and each of which will be used in a different way to feed our underlying systems to build the required functionality.As the programmers among you can likely guess, 'has' declares a field/member variable, 'can' declares a member function, and 'when' declares an event listener (a piece of code that will run in response to a particular 'event' that may occur on the object). The syntax of it all is just a sketch, so let's not concern ourselves too much with quibbling over whether 'has/can/when' is too cute / whether semicolons are the bane of mankind, etcThe point is that we have a portion of the mod that's using a very simple 'data' language to inform all of our heavy-duty systems of how to make inventories work. The 'has' statements inform our CTypes system about fields, address, alignments, etc. The 'can' is informing our more-general type system about the presence of callable member functions. The 'when's are informing our event handler system about events and their listeners.Then, of course, we must write some Lua to make it all work:And this looks more like what you would expect from standard Lua code. Here we're just implementing the functions referenced in the data section with standard Lua + some of the extra features added by the LT engine. Easy! All the 'bigger picture' elements that dictate when, where, and how a particular function runs are specified in the data section (e.g. addItem will be a method that can be called on any object with 'has Inventory', so that in other code we may say myShip:addItem(DenseTalvienite, 17) for example). Hence, much of the need for a modder to 'worry about' the internal mechanisms used to make LT performant (you know, the ones I talk about in pretty much every devlog ever?) is alleviated. Granted, they still exist and may still be interacted with at the code level, but for the majority of gameplay functionality, doing so should be unnecessary.The Lua mod loader that parses all the mods into a format that can then be shipped off to the appropriate engine-level systems is; what remains is to do the shipping-off bit and to move existing gameplay code into isolated mod files where appropriate. I'd like to have this done by the end of this weekIn response to anticipated concerns about 'oh no another language / LTSL / JOSH BE CAREFUL': Rest assured, the goal is entirely different here. I didn't define a new scripting language; this is just adata-description language that helps us connect all the moving pieces of our mod! It can be parsed with ~20 lines of Lua and the lovely string.gsub function. Anything more complex is unnecessary. I am not rabbitholing, I promiseThis bonus Josh-soliloquy is an optional part of your devlog experienceI've left out all talk of a 'breakthrough' that happened last week involving Lua data playing nicely with our C systems. It's technical and I've decided to err on the side of not-so-technical this week, but, briefly, I'll just say that this breakthrough has opened up a lot of fantastic augmentations to our existing systems, as well as tremendous new flexibility in splitting data & logic between Lua and C (using whichever makes the most sense for the task at hand). Previously, Lua->C interop was easy. LuaJIT's FFI makes it basically effortless. But going the other way? Accessing Lua-side data / calling Lua-side functions from C? It was not so easy when that data/function could be arbitrary and needed to live alongside supporting C data (think: sparse logic scheduler). Since Lua is GC'd, we can't get pointers to Lua data, and this is a major annoyance about which I've ranted before. However, there's a function that I overlooked until last week...hidden in the Auxiliary Library API...luaL_ref...and it solves all my problems. *Slams face vigorously for having overlooked this gem.* At least I know now.Until the 'morrow! (Well, actually I plan to do my next log on Friday. If I'm 'close but not quite' with respect to the mod system, then it will be Monday. Stay tuned!)PS ~ Sorry for lack of screenies, but most of this work wasn't amenable to it. Besides, you guys just got a whole boatload of them in Lindsey's last log!