Friday, August 18, 2017

Josh

Finished implementing zero-cost Lua wrappers for engine functionality (mentioned in last devlog; means we're getting superb performance when interacting with the LT Core)

Implemented enough of the foundational Lua code that we can all work on LT in Lua at the same time

Continued careful analysis of LuaJIT debug output; continued to ensure 100% JITing to assembly; continued to learn more about the trace compiler and how to beat it into submission

Hooked up our ECS to Lua + wrote the convenience wrappers that allow it to be used really easily (note: we now call it the 'CType' system instead of 'ECS', reason being that it is actually more general and flexible than an entity component system, although it can and does serve as one -- I'll explain more below!)

Ported Lua deferred renderer; modified to take advantage of all high-performance constructs

Made major improvements to our logarithmic z-buffer code while doing the above (this is the code that allows us to render objects at massive scales & distances without issue)

Implemented a work-in-progress event system in Lua for use in gameplay code

Implemented Lua wrappers for efficiently dealing with SDFs and OpenVDB (the library mentioned in the last log that we now use for extracting meshes from distance fields)

Implemented CSG functions on SDFs that are roughly 10x faster than OpenVDB's defaults, allowing complex, detailed meshes to be built very quickly (read: better procedural mesh algorithms coming soon!)

Wrote a simple pathtracer for fun since graphics Josh has been repressed Not totally irrelevant to LT, though, as we could perhaps use it for nice renders of procedural assets or for having a 'ground truth' reference for our physically-based rendering code

Not totally irrelevant to LT, though, as we could perhaps use it for nice renders of procedural assets or for having a 'ground truth' reference for our physically-based rendering code Built several Lua 'template' applications that allow us to easily various, specific pieces of functionality really quickly (just like the old LTSL sandboxes!)

Loads of other things that I'm forgetting

Adam

Started working in the Lua side of the LT codebase!

Put in the time to learn about all of LuaJIT's lovely 'idiosyncrasies'; Adam is now 100% capable of helping me deal with the technical side of LJ

Used LuaJIT magic to implement the first parametric type in our CType system -- the arraylist! This means that not only can we define and use new, native datatypes in Lua, but we can also have what amounts to native std::vector<T> and so forth of them. I know I've beat this horse to the ground, but seriously, the power we now have to create and use types that are as fast as compiled code simply can't be overstated! Memory management is by far the weakest spot in Lua's performance, and we are almost entirely side-stepping it.

Hooked up BSP code to Lua and used it to implement picking (being able to click on an object in the world and do something in response -- a seemingly-trivial task that is not at all trivial for large scenes and complex meshes, since it requires casting a ray through acceleration structures)

Got our first star system set up and working using CTypes and all the new high-perf Lua stuff!

Made a lot of improvements to our geometry code (BSPs, triangles, rays, etc.)

Implemented really fast BSP-Sphere intersection tests

Loads of other things that I'm forgetting (again)

Sean

High-Performance, High-Convenience: A Peak at Coding in LT

Native memory types can be defined on-the-fly in Lua; behind the scenes they are 100% as compact as the equivalent type would be in statically-compiled C; accessing the memory is extremely fast due to fancy, custom memory-pool allocation magic done automatically by the engine. All of this complexity is hidden from the user, so code remains very easy to read and write.

Types can be nested in other types, enabling 'component-based' design, as demonstrated with Health and Weapon. Of course, since this is a full-blown type system rather than an ECS alone, we can do things like have multiple health components, which ends up being useful sometimes!

'Member functions' can be attached to these types and called in the idiomatic Lua manner; convenience + performance = winning!

Generic functions like 'Weapon:damage' can be written without concern for the type of the target object and still 'just work,' despite the fact that we're using direct field access instead of table lookups like most dynamic languages (if you're a programmer and understand what I mean, that should blow your mind a bit )

) (Here's the even-more-mind-blowing magic) Functions that can operate on different types of objects (like Weapon:damage, which could be used to damage stations, hardpoints, etc) will be traced by LuaJIT and end up running as fast as equivalent statically-compiled constructs (in C this would require a switch statement or function pointer, in C++ this would require a template or virtual tables; here it requires nothing extra!!); this is a HUGE win for both performance and simplicity!

Although I didn't show it above, types can be built in a piece-meal fashion rather than defined all at once. In particular, mods have access to type definitions and can inject their own fields and methods as required to support extra, mod-related data and functionality (I showed something like this in a devlog a while back). As always, giving modders the ability to add just about anything conceivable to vanilla LT is really important to me!

Josh

Spoiler: SHOW Implemented stuff for better performance

Made it easier to write code while the game is running (I think)

Implemented more stuff for better performance

Hooked up pieces of code we've been working on for a while so we can actually use them, but then called it CType (instead of ECS) because that sounds cooler and makes more sense

Some stuff that printed stuff to screen that used to be in Lua is now in the C code core, so it's faster

Made major improvements to code that let us draw things that are far away (or very large)

Worked some on the foundation for gameplay code

Implemented stuff for better performance (Yes, again)

Implemented stuff for way better performance (No, really)

Got sidetracked and made some really shiny images with what seems to be a homebrewed renderer (which is particularly impressive, if not entirely LT-relevant, but it was done in Josh's free time)

Built some programs to let us test small pieces of LT Lua code quickly

Lots more where that came from

Adam

Spoiler: SHOW Started working on gameplay code and gameplay-related things!

Fully educated himself on Lua and LuaJIT, making him roughly on-par with Josh's level of accumulated LuaJIT knowledge

Long complicated bulletpoint that essentially means Adam is a wizard and is now quite near god status. Basically he implemented a lot of techy workaround stuff that overcomes Lua's inherent flaws. Neato, right?

More wizardry. Like Josh said, Adam implemented stuff that lets you click "on" an in-world object. Trickier than it sounds. It's taken care of now. It seems this may be more robust than the way Josh did it before, too.

Created a star system with the new LT code he and Josh have been working on. I basically copied this from what Josh said, it's pretty clear already.

Implemented more stuff for better performance (yes, Adam is guilty of this generic task description too)

Implemented some really performant collision-related tests, meaning we can check for collisions even faster. (Collisions are actually really hard for computers to detect. Our brains are much better suited to it.)

Loads of other stuff that Josh is forgetting.

Hey all!As usual, I apologize that these logs are doing a bit of temporal stretching lately -- honestly we're just really, really busy working on LT and it's tough to pull myself away these days, so consider it a good thing! When we last spoke, several major advances had set us up for fruitful weeks to come. I'm pleased to report that they were, indeed, very fruitful.At the moment, I can summarize our work by saying: we're dangerously close to shifting the majority of our efforts to sprinting on LT gameplay code in Lua. And we're doing everything we can to close the gap.Seriously, the 'road to (performant) gameplay' has been way longer than anyone wanted. But the end is in sight---Sadly, last Friday was Sean's final day here as an intern, since he's now headed off to college to study CS and become an unstoppable force of programming. We'll miss him for sure...his time here seemed way too short. Nonetheless, he was able to accomplish an impressive amount in the seemingly-little time that he had.In his final act of glory, Sean finished implementing a fully-generic, hierarchical, fleet-based AI maneuvering system, capable of representing fleets with any number of sub-fleets, and intelligently directing their motion. To demonstrate it, he showed us a group of several fleets flying together in a V formation, with each fleet comprised of squadrons in a V formation, each of which was comprised of fighters in a V formation...you get the idea! All of the math required to make it happen is rather difficult due to this arbitrary nesting requiring scale-invariance. Sean got it all hammered out and now we have proper, hierarchical fleets. I'm really excited to see this in-gameI don't have a gif on-hand, but I'll be sure to show off his work as soon as we attach it to the main LT Lua code.---As you all know, Adam and I have worked relentlessly over the past months to devise ways to achieve both extreme performance capable of handling the LT simulation logic, as well as the convenience of being able to write said logic in a language that won't rob us of every last drop of life. I'd like to show just a snippet of code demonstrating what our native type system ('CType' -- the final version of what we used to call the ECS) can do, how it looks, etc. So, have a slice of toy code! (Since it's harder to read without syntax highlighting, I just screenshotted some code in-editor)This is a massively-simplified battle simulator, intended only to showcase the simplicity of gameplay code. Nonetheless, it demonstrates some important features / fruits of our labor. Note: I cranked this out really quickly and didn't actually test it, there may be typos or slight mistakes---Well, once again I've completely and utterly failed to be briefIt's hard to be brief when there's so much happening and so much about which to be enthused. Speaking of being enthused. I must leave you all now to return to the codeHello, everyone! Talvieno here with a non-technical summary. There hasn't been much demand for one for a while, but this devlog seems to be worthy of it. Without further ado, let's get down to business! There are a couple "big things" to take away from this update - namely, Sean is no longer with us, and we're almost ready to start focusing entirely on gameplay. As to Josh's bullet lists, I'll break down what those mean in the spoilers below.(tl;dr: made things much faster and more performant)(tl;dr: Lots of coding wizardry)Okay, now we're to where Josh is telling us that Sean is leaving.We'll miss him dearly, and hope to see him around sometime in the future. In his final act of glory, he [technical gibberish that basically means he made ships in fleets fly around in an actually smart way, instead of all flying around haphazardly like in the LT Prototype]. It'll look pretty, believe me.As to all the last part (the image, the code, the final bulletpoint)... that can be summarized by saying Josh is proudly showing off the fruits of his labor, which unfortunately may not mean much to anyone that isn't a well-established programmer that can at least code on an intermediate level. The main things to take away from it is that it's fast, and flexible, meaning it's the perfect tool for modders. I'm actually understating it here - this is some top-tier wizardry that Josh and Adam have written up. If you're used to games running considerably more slowly when you start adding mods, that shouldn't be the case with LT. For those interested in writing their own mods, all this also means that you'll have a ridiculous amount of control over the functionality you can add to the game.And, that's it! Josh failed epically at being brief (does he ever not? We love him for it), but I've hopefully condensed it down to something a bit more pleasant to read for the non-technically inclined.I'll answer any questions as necessary, and thanks for reading!