Over the past week or so, there have been a few conversations on #developers about the ever-increasing time it takes to compile Firefox. Compiling Firefox used to be a relatively quick affair (well, on my machine, at least) and I’ve grown dissatisfied recently with builds taking ever-longer. All of the aforementioned discussions have boiled down to “well, we’re adding a lot of code and all that code takes longer to compile.”

Surely that can’t be the whole story. A mozilla-central build from a day or two took about 24 minutes on my machine (Linux x86-64 hyperthreaded 8-core @ 2.6GHz, srcdir on SSD, objdir on mechanical hard drive). We are not compiling 2.4x times more code than we were a year ago…or are we? Let’s look at some numbers.

I looked at my mozilla-central repository for some convenient tags to pick from and found the FIREFOX_AURORA_${REV}_BASE set of tags. I started compiling with mozilla-central tip, and then with FIREFOX_AURORA_24_BASE , and resolved to go as far back as I could without undue fiddling with things that wouldn’t compile. I made it to FIREFOX_AURORA_14_BASE before hitting compilation errors with moz_free. (19 and previous required small fiddling with js/src to set the right PYTHON , but that was easily scriptable.) That gives me 11 markers covering over a year of development.

I didn’t try compiling things multiple times to see how much the numbers fluctuated. I didn’t attempt to control for operating system disk caches or anything like that, just hg up -r $TAG , generated configure, ran configure --enable-optimize --disable-debug --disable-gstreamer (I haven’t bothered to get my gstreamer dependencies right on my development box) in a newly-created objdir, and recorded time for make -srj20 . The compiler was GCC 4.7 from Debian stable.

m-c tag real user sys 15 13m2.025s 83m36.229s 5m51.238s 16 13m40.359s 87m56.302s 6m15.435s 17 12m58.448s 94m21.514s 6m35.121s 18 14m21.257s 112m42.839s 7m36.213s 19 15m8.415s 120m48.789s 8m8.151s 20 16m26.337s 137m34.680s 9m11.878s 21 18m55.455s 165m29.433s 10m45.208s 22 19m27.887s 185m3.242s 11m53.853s 23 21m9.002s 203m46.708s 12m51.740s 24 21m51.242s 206m49.012s 12m55.356s tip 24m37.489s 219m27.903s 13m47.080s

Those 10-minute clobber build times were apparently the result of warm disk cache or my dreams. Or perhaps they’re due to an older compiler; I only started using GCC 4.7 recently and I’ve done some tests that show it’s a fair amount slower than GCC 4.4. Or maybe I was using a different build config then. Whatever the reason, we can see that compilation times have been increasing steadily, but can we correlate that to anything?

Let’s look at the number of C++, C, WebIDL, and IPDL source files. We include the last two because the conversion to WebIDL has been cited as a reason for increasing compilation times and IPDL because the IPDL compiler generates a number of source files as well (roughly three source files for every IPDL file). The C++ source files are split by extension; .cpp is the usual prefix to use inside Gecko, so separating .cpp and .cc gives some idea of how much outside C++ source code is being compiled. (Just counting source files is subject to issues, since we don’t compile everything in widget/ for every platform, but we’re counting all the source files therein as if they were. This is a rough estimate.)

m-c tag cpp cc c webidl ipdl 15 4060 577 2503 32 206 16 4117 1273 2893 34 213 17 4173 1278 2895 38 221 18 4468 1405 3123 61 227 19 4498 1408 3118 78 228 20 4586 1418 2779 169 228 21 4605 1392 2831 231 228 22 4979 1392 2980 305 228 23 5072 1393 2982 344 231 24 5101 1330 2983 413 234 tip 5211 1427 3029 436 234

First things first: we have a lot of code in the tree.

Now, there’s a couple of interesting things to point out. While there’s a lot of IPDL source files, the increase in IPDL can hardly be held responsible for the increase in compile time. So that’s one suspect out of the way. From 16 to 17, we barely increased the amount of code, but build times went down. This change might point to the recorded times being subject to quite a bit of variability.

We added a lot of WebIDL files from 20 to 21 while holding everything else constant and build times went up quite a bit! But we added roughly the same number of WebIDL files from 21 to 22 along with a lot more code and while build times went up, they didn’t go up by the same amount: 28 minutes of user time going from 20 to 21 and ~20 minutes of user time going from 21 to 22. There are a number of reasons why this could be so: we might not be compiling all those 500 C/C++ source files we added in 22 (ICU landed off by default in 22, so that’s ~400 files right there), the WebIDL files added in 21 might have had more of an impact, compilation-wise, the timings might fluctuate quite a bit, and so on and so forth. And notice that we added the same number of WebIDL files again from 23 to 24 and build times hardly changed. Sure, WebIDL means more files to compile. But I don’t think they are adding a disproportionate amount of time.

And finally, source code file counts. We had about 7800 source code files to compile back at Aurora 15 (remember, IPDL counts as three). We have about 10600 source files to compile now on trunk. That’s about a 40% increase that corresponds to roughly a doubling in time to compile. And yes, there’s also a lot of JS and other things that have been added that increase our time to compile. But I don’t think the amount of source code is the whole story.