The rate of checkins continues to rise rapidly. The volume of builds+tests run per checkin is also increasing rapidly. The combination of these two increases multiplied together is an impressive rate of growth.

How many hours of builds and tests do we run per commit?

254 compute hours = ~10.5 compute *days* (sep2013)

(sep2013) 137 compute hours = ~5.7 compute *days* (aug2012)

(aug2012) 110 compute hours = ~4.6 compute *days* (jan2012)

(jan2012) ~40 compute hours = ~1.6 compute *days*(2009)

10.5 compute *days* of builds and tests per checkin is a lot – especially when we average one checkin every ~6 minutes last month, and handle a checkin every 4minutes during peak times. Put another way, that’s an average of ~6.9 compute years of builds and tests being run per day, or 213 compute *years* for the month of August. Mind boggling.

If you look at the “wait times” posts in dev.tree-management, you’ll see that systems we run on AWS are scaling nicely to handle the load, and we continue to move production load over to AWS for nativefennec / B2G desktop builds / linux desktop …. and then recycle existing inhouse hardware to help with inhouse capacity for win32 and win64.

Some of this growth is good (new builds, new test suites).

But some of growth is bad (consistently failing tests; obsolete tests; builds that are no longer needed)… and this slows us all down.

You can help.