Metrics for Agile

“How can we tell how far along we are with our agile adoption?”

I heard this question again the other day.

Usually, the person who asks the question starts to answer it:

Number of teams using agile Number of people trained in agile Number of projects using agile Number of certified coaches.

Metrics like these won’t tell you what you need to know. More likely, they will lead you astray. How? Let me tell you a story.

Years ago, I worked for a company that was “installing” a Big Methodology from a Big Company. (The fact that they thought they were “installing” a methodology was probably the first warning sign.)

Every one in the department attended Big Methodology training. (This practice is sometimes called “Sheep Dip” training).

The VP mandated that all projects would use the Big Methodology.

The Installation Team audited to ensure that project managers and teams were complying and producing the required “work products” in accordance with the Required Work Products grid in the back of the very large Big Methodology binder.

Of course, there was some grumbling (from the people the Installation Team referred to as “Change Resisters.”) Eventually, people did comply. Every one went to training. Projects managers filled out the required templates, and checked the appropriate boxes. The metrics looked grand!

The VP declared, “Big Methodology is now business as usual!”

At the time, I scoffed at that statement. It was clear to me that people were not using Big Methodology, and that the promised benefits were nowhere in sight. The only things that had really changed were some check boxes and some names (documents became “work products” or “job aids,”).

But, now, I realize that the VP’s statement was TRUE!

We had Big Methodology, and things went on as they had–business as usual! Well, maybe a little worse because people were spending time producing the many documents specified on the Required Work Products grid.

The metrics the VP tracked were easy to count. But they only revealed surface compliance. They didn’t say anything about whether the organization was achieving the improvements promised by Big Methodology and hoped for by the VP.

So when you think about assessing how far along you are in your agile transformation, consider what you are trying to achieve.

I often suggest that managers track three metrics to understand how well their organization is functioning, and whether they are trending in the right direction.

The ratio of fixing work to feature work. How much time are people spending developing valuable new features vs. fixing stuff that wasn’t done right the first time? If you can figure out the sources of fixing work and make some progress there, you have a boost to productivity. Agile methods can address some of the sources of fixing work…but not all of them.

Cycle time. How long does it take to go from an idea to a valuable product in the hands of a customer? Again agile methods can help with delivery. But if it’s the upstream process–planning for products and releases–is broken, you may not see improvement until you address those issues, as well as the development process.

Number of defects escaping to production. This is a category of fixing-work that is a direct indicator that the quality of the development process is improving.

For each of these metrics, it is the trend that is important, not an absolute number. The trend will tell you if your attempts at improvement are having an effect. Remember, most changes take time to take hold. If the trend doesn’t move in a month, it may not mean you have taken the wrong action and need to change direction. If the trend isn’t moving over time, then, examine what is happening in the development area. But also look at other aspects of the system. There are few one-to-one cause and effect relationships in complex systems and the trend you see may or may not be directly related to your change. One company I worked with was alarmed to see that defects released to production went up after they started using agile methods. It turned out that prior to the effort to measure defects released to production, no one paid much attention unless the defect brought down a customer site. The increase in the defects trend was related to reporting, not a failure to improve quality.

I find that the three metrics above are generally useful for understanding how a software development organization is functioning as a system. But your reasons for adopting agile methods may be different. Consider the goals you are trying to achieve. What signals would tell you that you are moving in the right direction? How might you measure those? When you think about measures, be wary of target numbers. Measuring against targets almost always causes distortion. That means that people will behave so as to reach the target, perhaps in ways that are counter to the actual goal behind the target. Distortion will keep you from seeing the real picture, and may also cause real harm to your organization.

Useful metrics give you a window into how the system is functioning, and whether your change is having an effect. The numbers themselves are neither good nor bad. They are information that signals you to go and find out, investigate and reason about the system.