



{Disclaimer: These are our personal views they do not reflect our employer’s position. - Samir Khobragade, Dave Short}

I use the term ‘Developer Latency’ to describe the average time it takes for a programmer from saving their code changes and seeing them reflected in their development environment. This is an iterative process where an engineer is trying to code the algorithms they have in their mind. Quickly changing the code and seeing an expected result allows them to narrow down their assumption tree. They do this hundreds of times before that code is ready for a code review step which is followed by code check-in.

Over the last 2 decades, I have seen this latency range from 5 seconds to 5 minutes depending on the development environments and build systems and company processes. Four guys running a startup in the garage have a low Developer Latency as most of them are saving their changes in Eclipse, Android Studio, or Notepad++ and then running the program within 5-10 seconds. They are changing the JS, JSP, or ASPX file and refreshing the browser to see such changes reflected immediately. Or they are saving and stepping through their Java or C# code within 6-15 seconds. As companies grow, the use of heavy build systems, large dependency trees, restarting of servers, activating of environments, and auto running of unit tests increases average Developer Latency significantly. This time is the unaccounted delay in development which ultimately makes things go 50% to 100% slower within large companies.

I did an unofficial (and unscientific) survey of my 5 friends who worked or now work in the 5 different top tech companies. The Developer Latency number ranges from 30 seconds to 5 minutes. Following the ‘Rule of Five’ from the book How to Measure Anything by Douglas Hubbard, there is a 93.75% chance that the median of all the large tech companies’ population falls between those two numbers. I.e. these brilliant engineers run a series of build tools and sometimes wait minutes before they get to iterate on their code. I am sure looking at the breakdown of steps in-between, things will seem reasonable. It is easy to become complacent and accept this ‘reasonable’ state. New engineers join these companies and learn the same behavior from existing employees and accept it as a standard.

During my 10 years in a startup, a code change in a Notepad++ followed by a simple javac command or a refresh of the browser did the trick. We just tweaked tomcat containers to hot swap our changed classes. We ran all the unit-tests and code analysis only when we were ready for the code review. It reduced our developer latency significantly. I could go into the Subversion repository, checkout any project in Eclipse, add breakpoints, and step through any line I wanted. This reduced the ramp up times significantly for my new hires. Things seem to have gotten complicated with a new generation of highly optimized runtimes, app servers, and web servers.

Most of the “inner loop” Developer Latency is 1) unintentional and 2) addressable via technology fixes. Nobody focuses on optimizing these dev latency hotspots because the tools become “good enough,” so we all just live with it. There are other dilatory factors as well: the human/policy issues (Layers 9 and 10 in the OSI 7-Layer Model). They’re usually intended to provide some important trade-off benefits like protection of operational excellence or security requirements. We have come to believe that these kinds of delays are worth it as there can be a lot more at risk for a large company vs. the four guys in the garage.

The increased Developer Latency encourages engineers to reduce iterations and write more code before building and testing. In the ideal world, the opposite should be true. The programming thought process works ‘in the zone’. There are 100’s of variables, their scopes, TODOs, and commented blocks of code for testing that a programmer keeps in different corners of their brain. Anything that interrupts this zone kills productivity, making every task take longer. I believe speed and engineering excellence requirements are not mutually exclusive. We must achieve both. Even when the stakes are higher, the number of people involved increase, and the security and other operational processes become mandatory, it is possible to minimize this Developer Latency. At no point should we lower the bar for the quality, integration tests, load tests, unit tests, and continuous deployment. Instead we should raise the bar for our development and build tools.

Large tech companies attract the brightest talent in our industry. There is no reason the problem of Developer Latency can’t be solved in these companies. Being conscious of Developer Latency, measuring it, and optimizing build and development processes will get there. Till then the responsibility of fast innovations will continue to be handled by the smaller companies and startups. Large companies will continue to wonder why they can’t innovate as fast as startups. They will keep throwing twice the number of people at the projects to mitigate the issue.