Forward Compatiblity

Since Banished was shipped, when I go through the process of adding to or changing the game, I’m usually thinking about what might happen in the future. Will I add DLC? What form will it take? What are the implications of the mod kit? Will future changes break mods? Can mods break future DLC? Am I going to ship OSX and Linux builds? How do my current code and data changes effect those future desires?

So I find myself at a point where the mod kit is (sort of) close to complete. I need to make a few more code changes to really have good support for user changes and additions, but the overall ability of users to provide binary compiled assets to other users poses a problem.

The problem is that if people start making mods, and I then release an OSX or Linux version, those mods aren’t going to work on those platforms without the mod being recompiled.

Resources in Banished work a little bit like writing a program. When writing the code, usually you write some C or C++ code, compile it, and give end users a binary executable.

Resources for Banished are similar. If you make a model, a texture, an audio file, or general resource, you have an FBX, PNG, WAV, or text file as the user created resource. Those resources are compiled by a tool into a binary format that is fast to load and ready to use – the difference in load time between the game using the compiled version vs raw version is several seconds compared to several minutes. So shipping the raw resources really isn’t worth that wait time.

So what’s the problem? There’s currently a lot of binary graphics and audio data that is Windows only. OSX and Linux will use OpenGL, and there are no GLSL shaders stored in the game binary data. There’s only compiled DirectX byte code. And audio? It’s compressed and stored in XMA – XAudio’s format – similar to WMA.

If I were to just ship Linux and OSX at future date without the mod kit existing, I could patch the game data to be more cross platform, or even ship separate data – and no one would know a difference. But since there will most likely be mods out there – and in binary form, they won’t work on those platforms since key data doesn’t exist. The audio is in the wrong format, and there are no GLSL shaders.

Solutions? I know some companies are parsing compiled HLSL byte code and outputting GLSL. While this is a possibility, I’d like to use the actual features of GLSL and not be locked into what I can parse out of compiled byte code. Plus it makes sense for future projects to support it properly.

The audio side could be hard to deal with as well. Decoding XMA and reencoding to another audio format at load time isn’t something I want to get into.

I could just ship the mod kit, and new platforms just won’t work, and it will be up to mod authors to recompile their mod – and it may not just be a recompile. What if in implementing OpenGL, its better to modify some data for better compatibility? I’d probably do that. I like good code with good performance, instead of hacks to make things work. If the resource format changes, then mod authors will have to update any new materials they built. This really isn’t ideal.

I could have mods include the source data, and have newer releases include the entire toolset to recompile resources as needed, but again, that locks me into not drastically changing any file formats. Plus the raw data is a lot bigger. The Banished resource folder before compilation is around 700 megabytes – the shipped data is just over 100. Mods would have similar data bloat.

I could also put out a beta of the mod kit, let people get used to it while I take care of these cross platform issues, and then deprecate all binary mods. The mods would have to be recompiled and redistributed. This isn’t ideal for end users – especially if an author of a great mod stops maintaining it.

The best solution is really to make sure GLSL shaders are part of the data, and use an audio format that is usable on all three platforms.

So this weekend I’ve been looking at the feasibility of doing that.

I’ve been experimenting with OpenGL 3.2 and trying to get a sense of how long an implementation will take. It’s partially done already. If nothing else this will make sure I can build GLSL shaders using the material system and shaders already in use. Ideally the windows version will have a new OpenGL renderer that will be selectable just like DX9 and DX11.

On one hand this is great – From my graphics programmers background perspective, I’ll finally have a great test bed to really see which API runs the fastest (since this has been a debate from the beginning of time), since I can pick a scene and switch between rendering APIs at runtime. On the downside this increases development time before the mod kit appears. Although the DirectX implementations are only 44K(DX9) and 65K(DX11) of platform specific code so I assume GL will be similar. That little code shouldn’t take too long, right?

On the audio side FMOD seems capable of playing XMA samples, so switching over to that might not be so bad either, and if I do a little research to make sure it’ll work, it may not have to be done right away. Once I do port though, the API is available for all the platforms I want to support. Other cross platform audio libraries are out there – really i just have to decide on what format the audio will ship in.

On the upside of the extra development time to make sure mods work in the future, having OpenGL and FMOD (or other cross platform audio library) implemented will certainly bring me closer to having ports done as well. There’s not too many other platform specific parts of the engine. There’s file I/O which should be super easy, the math library, which is currently written in SSE – I just need a C++ port of it – also easy(ish), and input handling – reading from mouse, keyboard, and game controllers.

Now if I can stop falling asleep reading the OpenGL spec while trying to get the right triangles displayed on screen…