Have you heard the story of how the United States Air Force originally botched the cockpit in their planes?

In 1950 the U.S. Air Force took physical measurements of 4,000 pilots across 140 dimensions of size in order to inform airplane cockpit specs. That is, determine the average torso length or arm length and use that average to inform the placement of the cockpit seat, the yoke (i.e. the steering wheel), etc.

Seemed like a reasonable approach.

Only it turns out that the data-driven designed cockpit was a disaster. Good pilots were having uncontrolled crashes. Importantly, no one, including the pilots, quite understood what was amiss.

Is There Such a Thing as an Average Pilot?

Brigadier General Clinton D. “Casey” Vincent, early 1950s, near F-89 Scorpion fighter aircraft. (wiki)

Thankfully, one 23-year-old analyst — Lt. Gilbert S. Daniels — had some experience in measuring the human body and had a theory: perhaps there was no average pilot size.

To test his hypothesis, Daniels looked at the 10 most relevant physical dimensions of pilots from the study and created an “average pilot” based on the middle 30% range of the dimensions. Per Daniels’ analysis, the resulting “average pilot” ranged in height from 5’7″ to 5’11”. With this range in hand, Daniels then looked at the entire pool of 4,063 pilots and tried to match individual pilots to his “average pilot.”

Daniels was right: Not a single pilot fit the average. As the article so succinctly put it, “If you’ve designed a cockpit to fit the average pilot, you’ve actually designed it to fit no one.”

Armed with this insight the Air Force radically changed their design philosophy:

By discarding the average as their reference standard, the Air Force initiated a quantum leap in its design philosophy, centered on a new guiding principle: individual fit.

Rather than fitting the individual to the system, the military began fitting the system to the individual.

The Air Force had engineers immediately fix the problem. In short order components were made adjustable. And wouldn't you know it? The cockpit debacle is what ultimately led to adjustable car seats.

There is No ‘Average Man’

Why doesn’t the concept of the ‘average man’ work? Daniels summed it up:

“The tendency to think in terms of the ‘average man’ is a pitfall into which many persons blunder … It is virtually impossible to find an average airman not because of any unique traits in this group but because of the great variability of bodily dimensions which is characteristic of all men.”

Daniels’ insight — that there is no average human — could be characterized as a profound yet intuitive truth: people are (life is) complex. That complexity doesn’t do well with one-size-fits-all solutions.

Minding the Average When We Build Products

Typical web analytics are prone to the same “flaw of averages” Daniels saw in the Air Force. Just as with human physical dimensions, there is great variability in how individuals access and interact with the Internet. Consider the following variables:

Device variability (laptop, desktop, tablet, smartphone),

Intra-device variability (iOs or Android; Windows, OS X, Linux),

Intra-OS hardware variability (Over 24,000 different Android devices estimated),

Browser variability (Chrome, Firefox, Safari, others),

Screen resolution differences,

Window sizing preferences,

more …

Run the numbers on the above combinations (some of which could change for the same user within the same day or same website session!) and you get an unfathomably differentiated population of use-cases.

Visualizing Android device fragmentation via OpenSignal.

Meanwhile, these hardware and software variables are the "easy" ones to measure. They say little to nothing about what the user is actually trying to do on a qualitative basis on your site or app.

To See Beyond the Average

The best analytics platforms will adapt to and account for variations in hardware, software, and most importantly, users. A robust web analytics platform—the kind that can manage digital experience data—will work in such a way that it connects from the highest-level (E.g. user trends, KPIs, other metrics in the aggregate) all the way down to the individual users, capable of understanding their particular experiences just how they happened.

This kind of digital experience platform is hard to come by, but it’s exactly what we’re building at FullStory. Our goal is to understand users where they are; not where the average says they should be.

See also: Do User Metrics Make You Heartless?

Updated. Originally published March 9, 2017.