In Out of the Crisis, page 121, Dr. Deming wrote:

the most important figures that one needs for management are unknown or unknowable (Lloyd S. Nelson, director of statistical methods for the Nashua corporation), but successful management must nevertheless take account of them.

So what do you do then? I am a strong advocate of Deming’s ideas on management. I see understanding system thinking, psychology, the theory of knowledge and variation as the tools to use when you can’t get precise measures (or when you can).

Even if you can’t measure exactly what you want, you can learn about the area with related data. You are not able to measure the exact benefit of a happy customer but you can get measures that give you evidence of the value and even magnitude. And you can get measures of the costs of dis-satisfied customers. I just mention this to be clear getting data is very useful and most organizations need to focus on gathering sensible data and using it well.

Without precise measures though you must use judgment. Judgment will often be better with an understanding of theory and repeated attempts to test those theories and learn. Understanding variation can be used even if you don’t have control charts and data. Over-reaction to special causes is very common. Even without data, this idea can be used to guide your thoughts.

The danger is that we mistake measures for the thing itself. Measures are a proxy and we need to understand the limitation of the data we use. The main point Deming was making was we can’t just pretend the data we have tells us everything we need to know. We need to think. We need to understand that the data is useful but the limitations need to be remembered.

Human systems involve people. To manage human systems you need to learn about psychology. Paying attention to what research can show about motivation, fear, trust, etc. is important and valuable. It aids management decisions when you can’t get the exact data that you would like. If people are unhappy you can see it. You may also be able to measure aspects of this (increased sick leave, increased turnover…). If people are unhappy they often will not be as pleasant to interact with as people who are happy. You can make judgments about the problems created by internal systems that rob people of joy in work and prevent them from helping customers.

For me the key is to use the Deming’s management system to guide action when you can’t get clear data. We should keep trying to find measures that will help. In my experience even though there are many instances where we can get definite data on exactly what we want we fail to get data that would help guide actions a great deal). Then we need to understand the limitations of the data we can gather. And then we need to continually improve and continually learn.

When you have clear data, Deming’s ideas are also valuable. But when the data is lacking it is even more important to take a systemic approach to making management decisions. Falling back into using the numbers you can get to drive decision making is a recipe for trouble.

Related: Manage what you can’t measure – Statistical Engineering Links Statistical Thinking, Methods and Tools – outcome measures