Yesterday, my blog post featured a YouTube video that Everett Clinic was kind enough to share with the world, showing their Daily Huddle process (“Video of a Daily Huddle at Everett Clinic“).

In that post, I alluded to there being one thing in the video that I would want to improve upon and one commenter was sort of on the mark about what it was, or at least Karen was in the right general direction. Please, please, please don't take this post as a slam on Everett Clinic. I'm using this as a teaching example with a few suppositions and made-up scenarios building on their video.

Everett Clinic appeared to be reviewing metrics daily, which can be a good thing. But, it appears they're falling into an old trap from manufacturing: comparing performance to an arbitrary specification. Here comes my Dr. Deming soapbox… I need a picture of myself like this one… in this post I'll present a better alternative to comparing a single data point against a goal.

From the Daily Huddle video, we saw the leader say they only had a 5.x% call abandonment rate. The leaders seemed to pat themselves on the back that the results were OK because it was below their goal of a 10% abandonment rate.

I'm making up this visual, but you might see something like this hanging on the wall in a typical organization:

As Wheeler points out, comparing a single data point against a goal isn't very helpful in terms of managing and improving the system. If the rate was 5.x% today, what do we expect it to be tomorrow? How do we know if we're headed in the right direction? This two point comparison doesn't tell us anything particularly useful.

About the 10% goal…. who cares about the goal? Honestly. Who sets the goal? Management did. Why not 12%? Why not 3%? The goal is often somewhat arbitrary. Is it supposed to inspire people to get better (a “stretch goal”)? Sometimes the goal is rationalized and chosen through benchmarking, but what if the benchmark organizations have similar processes, systems, and results? Many people I know in healthcare value benchmarking less after being exposed to Lean, because they realize they were benchmarking against organizations that had the same waste they did.

I'd argue that customers don't care about goals or benchmarks. How do the customers feel about the system's performance? I bet 5.x% of them weren't happy that they couldn't get through to the clinic. Stretch goals might be frustrating (and unrealistic), but I might be a bit demoralized thinking the goal was something as high as 10% and we were easily hitting that every day (doesn't management think we can figure out how to do better?)

Please, don't get me wrong – I'm not implying that Everett Clinic doesn't care about the 5.x% rate or that they don't wish it could be lower. My point here is about the presentation of the data, it's not meant to be a broad criticism of them, but rather a general learning case. I see most organizations present performance measures the same way. Again, we can do better.

Manufacturers often fell (my optimistic past tense) into the trap of measuring their product quality against specs instead of things that truly mattered to the customer. Let's say you're drilling a hole that has to be within a certain size range – the lower and upper bounds are called specification limits or tolerances.

In any setting if you're not meeting specs, the easiest thing to do is… to change the specs!!! If a factory is having trouble meeting the tolerances (this often happened at one of my previous manufacturing employers), then plant management would often complaint that the specs should be wider and easier to hit.

The same could happen in a service setting. If an organization had a goal of 2.0% call abandonment and they were always at 5 or 6%, they could complain that the goal should be higher, or “more realistic” some might say. Again, that's all beside the point.

If Everett Clinic had a 5% “goal” for call abandonment, would they have reacted differently, with the 5.x% number being above that goal?

What matters is working toward a goal of zero call abandonment. Looking at 5.x% and comparing it against an arbitrary goal does little to tell us about the health of the work system. Is 5.x% the typical average performance? Is that much higher than usual?

This is a great opportunity to use the methods of Statistical Process Control. The main management decision is to decide “react” or “not react” to that daily data point. SPC helps us with this (again, Wheeler's brilliant little book explains this far better than I can in a blog post).

If we choose “not react” because 5.x% is lower than the goal, we might be missing an opportunity for process improvement. Generally, it's better to present more than one data point – even if you don't do full-blown SPC, you should present a “run chart.' See this chart that I made up (a simplified run chart, not a full SPC chart):

In the days leading up to the 5.X% number, the history would have told us to expect a rate between 8 and 10% — it looks like the definition of a system that is “in control,” statistically speaking.

In my made up example, the day with 5.X% is clearly outside of the normal range – there is likely some “special cause” variation that occurred. If we simply said “OK, better than goal” and moved on, what are we missing out on?

Maybe something happened differently in the system – did somebody make a process change? What happened? That's important to figure out — so, in a good, way the right response in my scenario would be “REACT.” What changed in the system and how do we celebrate and capture that day's system so it's our system going forward (until we find a new way!).

If that were made the new process, the chart might look like this over the next month:

In this case, that 5.X% established the start of a new stable process where we'd rightfully expect to have an abandonment rate between 4 and 6% every day. In this new system, a single day of 5.X% would lead to us “not reacting” because there is no special cause – that day's performance would be due to common cause variation. Finding and attacking that common cause variation is a different thought process than “what happened yesterday?” and it's certainly different than a blaming response of “who messed up yesterday?”

Back to the first chart, where the 5.x% showed an improvement to react to, the same thinking would hold true in the opposite case, as shown here:

In this case, although the 5.x% is still lower than the goal, the manager SHOULD react. “But it's still in spec,” someone might say… but, no, it's a likely “special cause,” this time in the bad direction. What changed in the system this day? We don't ask, whose fault it is. What can you do to make sure that was truly an anomaly? Then, the next question is to ask what you can to make sure you detect that problem (if it happens again) BEFORE the end of the day when the data is tabulated. How can you have real-time problem detection?

So, to wrap up – what matters is NOT our performance for a day versus a goal. Goals are pretty irrelevant, at least in terms of making daily management decisions. What DOES matter is looking at our day's performance in the context of a run-chart or SPC history. I think that's the best way to manage daily data like this.

And that's easy to do. At a recent hospital gemba visit, I saw daily run charts being maintained by front-line staff with a pen, updating the chart manually each day. Using run-charts and managing that way doesn't require daily printing of an Excel chart (although I would personally keep a history of the data in Excel).

What are your experiences with this type of approach to management data?

Please post a comment and join the discussion. Subscribe to get notified about posts via email daily or weekly.