Back by popular demand, Haelo’s Data Coordinator, Nick John, from the Measurement team tells us about the need for data.



When working in improvement you often see before and after surveys or audits used as evidence that a project has been successful. This is often displayed in a data table or a bar chart and can be crucial to a project receiving further funding or being allowed to continue. However does this method of measurement really give us enough information for these sorts of decisions to be made. Can a before and after audit really show us if a system has changed or improved?

Take this example above; we’ve been working to reduce the number of bad things occurring in our system between Apr-14 and Apr-15. This chart seems to suggest that the project has been a success. The number of bad things occurring in Apr-15, after our project, was less than the number of bad things occurring in Apr-14, before we started our improvement work. However, can we really tell from this chart if we’ve seen an improvement?

If we take the exact data from the previous bar chart and plot it in a run chart, we can see the amount of information that we are missing. What happened in the 11 months between these two data points while we were running our improvement project?

Maybe the data between could look like this, which definitely seems to show a reduction, with something we’ve put into place being particularly effective in Sep-14.

However the data in between those two points could also look like this. Maybe that first month we measured was just a particularly bad month and we usually have much less bad things occurring.

Maybe the missing data looks like this? Perhaps the last point was just a particularly good month and we usually have a much higher number of bad things occurring.

Maybe the missing data is a lot more varied than we expected and shows a pattern like this. This seems to indicate that our project hasn’t made a whole lot of progress in terms of changing the system and reducing the number of bad things occurring.

The major issue with a before and after measurement system, when working in improvement, comes from the comparison of two numbers. If you compare any two numbers, one is always likely to be higher than the other. This can often lead you to false conclusions about the success of a project. A much more useful method when trying to understand if a system has improved is to measure often and measure regularly.

Plotting your data over time gives you the ability to understand how your system is performing in real time and spot signs of change as the points begin to track up or down. If you have annual data, try looking at it monthly, if you have monthly data try looking at it weekly. The more data points you plot the better chance you have to identify trends, patterns or statistical signs of improvement.

Your reactions on twitter

The power of run charts in healthcare – an excellent synopsis https://t.co/fC4Vyvk17i — Ryan McGuire (@ryan_a_mcguire) May 13, 2016

Sunday morning seems to be data morning. More on the advantages of run charts. #HQA #SPSP https://t.co/e6KZ9KqnmL — Highland QI (@HighlandQI) May 1, 2016