In a world awash with big data and struggling with the questions it poses, a new history of our statistical foundations could hardly be more timely

A 15th-century manuscript shows surveyors measuring boundaries The Art Archive / Bibliothèque Inguimbertine Carpentras / Gianni Dagli Orti

NEVER entrust measurement to a monarch. With the arrival of a smaller sovereign, you might see your property literally shrink in line with their shoe size. To avoid this injustice, medieval surveyors came up with an ingenious idea. They would line up 16 citizens, measure the combined length of their feet, and then divide the total into 16 equal foot-long segments. While they didn’t formalise the mathematics, it was an early example of a powerful technique for analysing data that statisticians now call aggregation.

According to Stephen M. Stigler, aggregation is one of the “seven pillars of statistical wisdom”, principles such as probability and information measurement he characterises as the “disciplinary foundation” of statistics. His engaging book incisively (albeit rather technically) explores their history, in search of “a unity at the core of Statistics both across time and between areas of application”.

The historical development of aggregation exemplifies the tortuous path to unification. In ancient times, the Athenian general and historian Thucydides described an attempt by soldiers to estimate the height of a wall before a siege. The calculation was made by counting rows of bricks. Though “some [soldiers] might miss the right calculation,” he wrote, “most would hit upon it”. Making siege ladders based on the most often-arrived-at number, the soldiers showed an intuitive grasp of the statistical average now known as the mode.

Mode and mean averages may have been in everyday use, but they were erratically applied by scholars until at least the 1600s. Researchers each used their own recipe for aggregation. Inconsistency and ambiguity were rampant, and serious impediments to the accumulation of knowledge.

One reason for this lack of consensus was surely the paucity of communication between scholars working in relative isolation. But there may also have been some hesitation due to the counter-intuitive nature of aggregation. After all, as Stigler observes, averaging is a radical idea: “you can actually gain information by throwing information away”. Observers’ identities are discarded, which means no observation holds more weight, even if the observer has higher credibility. In order to embrace this simplest principle of statistics, scientists had to overcome social conventions and common sense.

“Averaging is a radical idea: you can actually gain information by throwing information away“

Other “pillars” have different histories, but they have this quality of radicalism in common. Information measurement, for example, is also counter-intuitive because the information gained doesn’t increase linearly with the number of observations made; to double your accuracy, you need to quadruple your effort.

Even Isaac Newton seems not to have appreciated this. While he was Warden of the London Mint, coins were weighed in batches to assess them for consistency: the weight deviation permitted per coin was multiplied by the quantity in the lot. As a result, quality control was an order of magnitude looser than intended. A shrewder (and less scrupulous) warden could have made a fortune playing the statistics.

Stigler is emphatic that the failings of the past shouldn’t make us smug. On the contrary, he argues convincingly that we still need to apply statistical methods with care, especially when negotiating big data: “With ever larger data sets come more questions… and more worry that the flexibility inherent in modern computation will exceed our capacity to calibrate, to judge the certainty of our answers.”

Stigler doesn’t elaborate on this – except to allude to the need for an “eighth pillar” – and his book is too modular to articulate the promised unity that might lead us beyond his traditional seven. That said, this lively account of a radically counter-intuitive past at least encourages us to question big data’s reputation. Never entrust measurement to a monarch – or judgement to a computer.

The Seven Pillars of Statistical Wisdom Stephen M. Stigler Harvard University Press

This article appeared in print under the headline “Of monarchs and measurement”