Doom3 Source Code Review: Introduction (Part 1 of 6) >>

On November 23, 2011 id Software maintained the tradition and released the source code of their previous engine. This time is was the turn of idTech4 which powered Prey, Quake 4 and of course Doom 3. Within hours the GitHub repository was forked more than 400 times and people started to look at the game internal mechanisms/port the engine on other platforms. I also jumped on it and promptly completed the Mac OS X Intel version which John Carmack kindly advertised.





In terms of clarity and comments this is the best code release from id Software after Doom iPhone codebase (which is more recent and hence better commented). I highly recommend everybody to read, build and experiment with it.





Here are my notes regarding what I understood. As usual I have cleaned them up: I hope it will save someone a few hours and I also hope it will motivate some of us to read more code and become better programmers.



Part 1: Overview

Part 2: Dmap

Part 3: Renderer

Part 4: Profiling

Part 5: Scripting

Part 6: Interviews (including Q&A with John Carmack)



From notes to articles...

I have noticed that I am using more and more drawing and less and less text in order to explain codebase. So far I have used gliffy to draw but this tool has some frustrating limitations (such as lack of alpha channel). I am thinking of authoring a tool specialized in drawing for 3D engines using SVG and Javascript. I wonder if something like this already exist ? Anyway, back to the code...

Background

Getting our hands on the source code of such a ground breaking engine is exciting. Upon release in 2004 Doom III set new visual and audio standards for real-time engines, the most notable being "Unified Lighting and Shadows". For the first time the technology was allowing artists to express themselves on an hollywood scale. Even 8 years later the first encounter with the HellKnight in Delta-Labs-4 still looks insanely great:



Your browser does not support the video tag.

First contact

The source code is now distributed via Github which is a good thing since the FTP server from id Software was almost always down or overloaded.



The original release from TTimo compiles well with Visual Studio 2010 Professional. Unfortunately Visual Studio 2010 "Express" lacks MFC and hence cannot be used. This was disappointing upon release but some people have since removed the dependencies.





Windows 7 : =========== git clone https://github.com/TTimo/doom3.gpl.git





For code reading and exploring I prefer to use XCode 4.0 on Mac OS X: The search speed from SpotLight, the variables highlights and the "Command-Click" to reach a definition make the experience superior to Visual Studio. The XCode project was broken upon release but it was easy to fix with a few steps and there is now a Github repository by "bad sector" which works well on Mac OS X Lion.





MacOS X : ========= git clone https://github.com/badsector/Doom3-for-MacOSX-

Notes : It seems "variable hightlights" and "Control-Click" are also available on Visual Studio 2010 after installing the Visual Studio 2010 Productivity Power Tools. I cannot understand why this is not part of the vanilla install.



Both codebases are now in the best state possible : One click away from an executable !

Download the code.

Hit F8 / Commmand-B.

Run !

Trivia : In order to run the game you will need the base folder containing the Doom 3 assets. Since I did not want to waste time extracting them from the Doom 3 CDs and updating them: I downloaded the Steam version. It seems id Software team did the same since the Visual Studio project released still contains "+set fs_basepath C:\Program Files (x86)\Steam\steamapps\common\doom 3" in the debug settings!



Trivia : The engine was developed with Visual Studio .NET (source). But the code does not feature a single line of C# and the version released requires Visual Studio 2010 Professional in order to compile.

Trivia : Id Software team seems to be fan of the Matrix franchise: Quake III working title was "Trinity" and Doom III working title was "Neo". This explains why you will find all of the source code in the neo subfolder.

Architecture

The solution is divided in projects that reflect the overall architecture of the engine:





Projects Builds Observations Windows MacO SX Game gamex86.dll gamex86.so Doom3 gameplay Game-d3xp gamex86.dll gamex86.so Doom3 eXPension (Ressurection) gameplay MayaImport MayaImport.dll - Part of the assets creation toolchain: Loaded at runtime in order to open Maya files and import monsters, camera path and maps. Doom3 Doom3.exe Doom3.app Doom 3 Engine TypeInfo TypeInfo.exe - In-house RTTI helper: Generates GameTypeInfo.h : A map of all the Doom3 class types with each member size. This allow memory debugging via TypeInfo class. CurlLib CurlLib.lib - HTTP client used to download files (Staticaly linked against gamex86.dll and doom3.exe). idLib idLib.lib idLib.a id Software library. Includes parser,lexer,dictionary ... (Staticaly linked against gamex86.dll and doom3.exe).

Like every engine since idTech2 we find one closed source binary (doom.exe) and one open source dynamic library (gamex86.dll).:





Most of the codebase has been accessible since October 2004 via the Doom3 SDK: Only the Doom3 executable source code was missing. Modders were able to build idlib.a and gamex86.dll but the core of the engine was still closed source.



Note : The engine does not use the Standard C++ Library: All containers (map,linked list...) are re-implemented but libc is extensively used.



Note : In the Game module each class extends idClass. This allows the engine to perform in-house RTTI and also instantiate classes by classname.



Trivia : If you look at the drawing you will see that a few essential frameworks (such as Filesystem ) are in the Doom3.exe project. This is a problem since gamex86.dll needs to load assets as well. Those subsystems are dynamically loaded by gamex86.dll from doom3.exe (this is what the arrow materializes in the drawing). If we use a PE explorer on the DLL we can see that gamex86.dll export one method: GetGameAPI :







Things are working exactly the way Quake2 loaded the renderer and the game ddls: Exchanging objects pointers:



When Doom3.exe starts up it:

Loads the DLL in its process memory space via LoadLibrary .

. Get the address of GetGameAPI in the dll using win32's GetProcAddress .

in the dll using win32's . Call GetGameAPI .

gameExport_t * GetGameAPI_t( gameImport_t *import );

At the end of the "handshake", Doom3.exe has a pointer to a idGame object and Game.dll has a pointer to a gameImport_t object containing additional references to all missing subsystems such as idFileSystem .



Gamex86's view on Doom 3 executable objects:

typedef struct { int version; // API version idSys * sys; // non-portable system services idCommon * common; // common idCmdSystem * cmdSystem // console command system idCVarSystem * cvarSystem; // console variable system idFileSystem * fileSystem; // file system idNetworkSystem * networkSystem; // network system idRenderSystem * renderSystem; // render system idSoundSystem * soundSystem; // sound system idRenderModelManager * renderModelManager; // render model manager idUserInterfaceManager * uiManager; // user interface manager idDeclManager * declManager; // declaration manager idAASFileManager * AASFileManager; // AAS file manager idCollisionModelManager * collisionModelManager; // collision model manager } gameImport_t;

Doom 3's view on Game/Modd objects:

typedef struct { int version; // API version idGame * game; // interface to run the game idGameEdit * gameEdit; // interface for in-game editing } gameExport_t;

Notes : A great resource to understand better each subsystems is the Doom3 SDK documentation page: It seems to have been written by someone with deep understanding of the code in 2004 (so probably a member of the development team).

The Code

Before digging, some stats from cloc :



./cloc-1.56.pl neo 2180 text files. 2002 unique files. 626 files ignored. http://cloc.sourceforge.net v 1.56 T=19.0 s (77.9 files/s, 47576.6 lines/s) ------------------------------------------------------------------------------- Language files blank comment code ------------------------------------------------------------------------------- C++ 517 87078 113107 366433 C/C++ Header 617 29833 27176 111105 C 171 11408 15566 53540 Bourne Shell 29 5399 6516 39966 make 43 1196 874 9121 m4 10 1079 232 9025 HTML 55 391 76 4142 Objective C++ 6 709 656 2606 Perl 10 523 411 2380 yacc 1 95 97 912 Python 10 108 182 895 Objective C 1 145 20 768 DOS Batch 5 0 0 61 Teamcenter def 4 3 0 51 Lisp 1 5 20 25 awk 1 2 1 17 ------------------------------------------------------------------------------- SUM: 1481 137974 164934 601047 -------------------------------------------------------------------------------



The number of line of code is not usually a good metric for anything but here it can be very helpful in order to assess the effort to comprehend the engine. 601,047 lines of code makes the engine twice as "difficult" to understand compared to Quake III. A few stats with regards to the history of id Software engines # lines of code:



#Lines of code Doom idTech1 idTech2 idTech3 idTech4 Engine 39079 143855 135788 239398 601032 Tools 341 11155 28140 128417 - Total 39420 155010 163928 367815 601032



Note : The huge increase in idTech3 for the tools comes from lcc codebase (the C compiler used to generate QVM bytecode) .

Note : No tools are accounted for Doom3 since they are integrated to the engine codebase.



From a high level here are a few fun facts:



For the first time in id Software history the code is C++ instead of C. John Carmack elaborated on this during our Q&A.



Abstraction and polymorphism are used a lot across the code. But a nice trick avoids the vtable performance hit on some objects.

All assets are stored in human readable text form. No more binaries. The code is making extensive usage of lexer/parser. John Carmack elaborated on this during our Q&A.

Templates are used in low level utility classes (mainly idLib) but are never seen in the upper levels so they won't make your eyes bleed the way Google's V8 source code does.

In terms of code commenting it is the second best codebase from id software, the only one better is Doom iPhone, probably because it is more recent than Doom3. 30% comments is still outstanding and find it rare to find a project that well commented! In some part of the code (see dmap page) there are actually more comments than statements.

OOP encapsulation makes the the code clean and easy to read.

The days of low level assembly optimization are gone. A few tricks such as idMath::InvSqrt and spacial localization optimizations are here but most of the code just tries to use the tools when they are available (GPU Shaders, OpenGL VBO, SIMD, Altivec, SMP, L2 Optimizations ( R_AddModelSurfaces per model processing)...).

It is also interesting to take a look at idTech4 The Coding Standard (mirror) defined by John Carmack (I particularly appreciated the comments about const placement).

Unrolling the loop

Here is the main loop unrolled with the most important parts of the engine:



idCommonLocal commonLocal; // OS Specialized object idCommon * common = &commonLocal; // Interface pointer (since Init is OS dependent it is an abstract method int WINAPI WinMain( HINSTANCE hInstance, HINSTANCE hPrevInstance, LPSTR lpCmdLine, int nCmdShow ) { Sys_SetPhysicalWorkMemory( 192 << 20, 1024 << 20 ); //Min = 201,326,592 Max = 1,073,741,824 Sys_CreateConsole(); // Since the engine is multi-threaded mutexes are initialized here: One mutex per "critical" (concurrent execution) section of code. for ( int i = 0; i < MAX_CRITICAL_SECTIONS; i++ ) { InitializeCriticalSection( &win32.criticalSections[i] ); } common->Init( 0, NULL, lpCmdLine ); // Assess how much VRAM is available (not done via OpenGL but OS call) Sys_StartAsyncThread(){ // The next look runs is a separate thread. while ( 1 ){ usleep( 16666 ); // Run at 60Hz common->Async(); // Do the job Sys_TriggerEvent( TRIGGER_EVENT_ONE ); // Unlock other thread waiting for inputs pthread_testcancel(); // Check if we have been cancelled by the main thread (on shutdown). } } Sys_ShowConsole while ( 1 ){ Win_Frame(); // Show or hide the console common->Frame(){ session->Frame() // Game logic { for ( int i = 0 ; i < gameTicsToRun ; i++ ) RunGameTic(){ game->RunFrame( &cmd ); // From this point execution jumps in the GameX86.dll address space. for ( ent = activeEntities.Next(); ent != NULL; ent = ent->activeNode.Next() ) ent->GetPhysics()->UpdateTime( time ); // let entities think } } session->UpdateScreen( false ); // normal, in-sequence screen update { renderSystem->BeginFrame idGame::Draw // Renderer front-end. Doesn't actually communicate with the GPU !! renderSystem->EndFrame R_IssueRenderCommands // Renderer back-end. Issue GPU optimized commands to the GPU. } } } }



For more details here is the fully unrolled loop that I used as a map while reading the code.





It is a standard main loop for an id Software engine. Except for Sys_StartAsyncThread which indicate that Doom3 is multi-threaded. The goal of this thread is to handle the time-critical functions that the engine don't want limited to the frame rate:

Sound mixing.

User input generation.

Trivia : idTech4 high level objects are all abstract classes with virtual methods. This would normally involves a performance hit since each virtual method address would have to be looked up in a vtable before calling it at runtime. But there is a "trick" to avoid that. All object are instantiated statically as follow:

idCommonLocal commonLocal; // Implementation idCommon * common = &commonLocal; // Pointer for gamex86.dll

Since an object allocated statically in the data segment has a known type the compiler can optimize away the vtable lookup when commonLocal methods are called. The interface pointer is used during the handshake so doom3.exe can exchange objects reference with gamex86.dll but in this case the vtable cost is not optimized away.



Trivia : Having read most engines from id Software I find it noticeable that some method name have NEVER changed since doom1 engine: The method responsible for pumping mouse and joystick inputs is still called: IN_frame() .



Renderer

Two important parts:

Since Doom3 uses a portal system, the preprocessing tool dmap is a complete departure from the traditional bsp builder. I reviewed it to the deep down on a dedicated page.











is a complete departure from the traditional bsp builder. I reviewed it to the deep down on a dedicated page. The runtime renderer has a very interesting architecture since it is broken in two parts with a frontend and backend: More on the dedicated page.



Profiling

I used Xcode's Instruments to check where the CPU cycle were going. The results and analysis are here.

Scripting and Virtual Machine

In every idTech product the VM and the scripting language totally changed from the previous version...and they did it again: Details are here.

Interviews

While reading the code, several novelties puzzled me so I wrote to John Carmack and he was nice enough to reply with in-depth explanations about:



C++.

Renderer broken in two pieces.

Text-based assets.

Interpreted bytecode.

I also compiled all videos and press interviews about idTech4. It is all in the interviews page.



Recommended readings

As usual a few books that you may enjoy if you enjoy the code:





One more thing

Summer is coming and it was not always easy to focus...











...but overall it was a blast to read most of it. Since idTech5 source code will not be released anytime soon (if ever) this leaves me with idTech3 (Quake III) not yet reviewed. Maybe I will write something about it if enough people are interested.

@