I only have data about myself, of course, but I’m very curious about other developers. Are there developers with little variance, and each day they work on their main project consistently close to some average?

Until I started collecting data, I thought I was one of those developers. But it turns out, I am not. And if you think you are, maybe you aren’t either. In fact, maybe yours has even more variance! It’s hard to know. I look at developers around me, and suspect that about half of them probably have more unpredictable schedules!

After all, don’t we all have questions to answer, code to review, people to talk to, production fires to fight?

Simulating project length

OK here’s the good part.

Suppose I’m given a project, and I think about it and decide it will take 2000 minutes to complete. On average, I know I do 200 minutes per day on a main project, so the expected length of the project is therefore 10 days, give or take a few depending on luck.

I can use my data to simulate how long the project will actually end up taking. And I can simulate it over and over, to see how often and how much it varies from the expected length.

(Remember, we’re still “assuming” I’m a god developer and all the other random variables don’t exist.)

Here’s an example run of the sim: On day 1 of the project, we randomly select a day from my dataset, and take its “number of minutes on my main project”. OK, we got 150 minutes. Have we reached 2000 total minutes yet? No? Then go on to day 2. Again, we randomly select a day from my dataset. OK, we got 300 minutes. Have we reached 2000 minutes yet? No, only 450 so far. …And so on. Eventually, halfway through the 8th day, we reach 2000 minutes. Therefore on this run of the simulation, it took 7.5 days to finish the project (we only needed half of day 8).

That was one run, but what we really want to do, is record that we finished on “Day 8” in a histogram, and then do 1 million more runs of the simulation, and record each result in the histogram.

Hopefully you can see the intuition of why this simulation works: how much “main project” work I do in a day is a random variable, but my collected data captures the distribution of that random variable. If I have a lot of “0 minute” days, then my dataset will reflect that, and so the simulation will randomly select 0 minute days often. If my collected days were pretty consistent, then the simulation would randomly select days that are consistent.

Simulation result

Anyway, here is the resulting histogram for simulating this 10-day project using my unpredictable schedule, and finding out how long it could actually take:

The distribution of 1 million simulated runs of a 10-day project. The runs simulate the effect of the variance of my daily schedule, and so they vary significantly from 10 days. How to read the graph: the bar labelled “13” has height “8.4%”, which means that 8.4% of the runs finished during day 13 of the project.

Gorgeous! Right? Here are some examples of fun observations to make.

60.1% of runs are ±1 or more days away from the expected 10 days.

95% of runs take between 7.0 and 14.5 days, to complete.

That last observation is also known as the “Two-tailed 95% confidence interval” of my project’s length.

Initial reaction

So, are you surprised? Or does it look about right to you?

For me, I was surprised. I definitely thought the variance of project length was wide, sure. Wider than this, even. However, I mostly think about sources of variance like “unexpected difficulties”, “unclear requirements”, that sort of thing. I never thought about an “unpredictable schedule” mattering much.

And yet, the project variance is already this huge without including those obvious other sources of variance! If we were able to include those as well, this graph would be much wider still!

Many software developers who need to report estimates, prefer not to report the expected average length (10 days), and instead prefer to give the 95% confidence interval (7.0 to 14.5 days). The above graph and observations probably demonstrate why: the estimate of 10 days really doesn’t tell us much, if it’s incorrect by 1.0+ days, in 60.1% of cases! Perhaps these developers have a good intuition for the vast randomness of software development.

But just look at the size of that 95% confidence interval, it’s 7.0 to 14.5 days of work! If you give an estimated range like that, there are people who will give you dirty looks. I reckon that many people get shocked that the second number is two times the first number, which seems like 100% error, or something like that. They aren’t visualizing the bell curve shape and tails.

Perhaps that is why many software developers prefer to just report the right-end of the interval (14.5 days). The truth can be scary.

And yet, in reality, with all the other random variables, this graph would probably be much wider. Maybe absurdly wide…it makes me think there is no such thing as estimating, really…

It makes me think, that there is no such thing as “being fairly sure a project will finish by X date”. There is only setting a reasonable limit, and working up until then. If you do finish before that, great, you got lucky. If you don’t, that’s common, just try to have something working reasonably well when you hit the negotiated limit…because otherwise, you might be working shockingly longer than you ever wanted to.

That’s just my reaction though, maybe yours is different! Think about it!

An important takeaway

There’s an important thing I think this experiment shows nicely.

When a project is far off from its estimate, it is often blamed on the developer’s estimation skill — if they were better at estimating, they would have been closer to predicting the real completion date.

But in this experiment, we’re taking estimation skill entirely out of the picture. We’re taking all skills out of the picture! We’re guaranteeing the actual project length in minutes is known and constant. 2000 minutes. And yet, we can see, the error can be quite large anyway! Because there is inherent randomness in software development.

When you roll a six-sided die, and you give an estimate of 3.5, and you end up rolling a 6, that doesn’t mean you estimated poorly. You estimated perfectly, 3.5 really is the average expected value. There is just inherent randomness in the act of rolling a six-sided die. You should be blaming that inherent randomness.

You can claim after the fact “well you should have known the die would bounce that way, I see no randomness, just ignorance”, but similarly, you can claim “well you should have known the the project would go that route, I see no randomness, just ignorance”. Both statements are equally unwise.

Even the god of software development cannot guarantee the actual project length will match the expected project length. There is inherent randomness in software development. For many of the random variables, it is hard to prove that the randomness is innate and unremovable, although nearly any software developer knows the intense randomness exists. But the “unpredictable schedule” random variable is a really useful one to study, since it’s so plainly visible, and so easy to show how much it affects project length.

Estimation skill probably does exist, but it is only one of many random variables, and there is no reason to blame estimation skill any more than the other random variables, when a project’s actual length differs from the estimated.