Tristan Caplan's Tetlockian Glossary By Bryan Caplan

This summer, I assigned Tetlock and Gardner’s Superforecasting (see here, here, and here) to my homeschoolers. They took to this masterpiece of political psychology like fish to water. I’d already taught them about human beings’ wacky mapping from language to quantitative probabilities, but these passages have really stuck with them:

In March 1951 National Intelligence Estimate (NIE) 29-51 was published. “Although it is impossible to determine which course of action the Kremlin is likely to adopt,” the report concluded, “we believe that the extent of [Eastern European] military and propaganda preparations indicate that an attack on Yugoslavia in 1951 should be considered a serious possibility.” …But a few days later, [Sherman] Kent was chatting with a senior State Department official who casually asked, “By the way, what did you people mean by the expression ‘serious possibility’? What kind of odds did you have in mind?” Kent said he was pessimistic. He felt the odds were about 65 to 35 in favor of an attack. The official was started. He and his colleagues had taken “serious possibility” to mean much lower odds. Disturbed, Kent went back to his team. They had all agreed to use “serious possibility” in the NIE so Kent asked each person, in turn, what he thought it meant. One analyst said it meant odds of about 80 to 20, or four times more likely than not that there would be an invasion. Another thought it meant odds of 20 to 80 – exactly the opposite. Other answers were scattered between these extremes. Kent was floored.

Ideally, analysts would adopt standard conventions on the correct way to translate from English to math. But since that won’t happen anytime soon, my son Tristan proposed a simple, effective half-measure. Namely: every serious book (and perhaps every article) should include a Probability Glossary in the opening pages, alongside the List of Illustrations and List of Tables. In these glossaries, each author would explicitly state what probabilities (or probability ranges) he assigns to probability-relevant English words. For example, here are roughly the probabilities I have in mind when writing this blog:

English Term My Probability Absolutely certain 100% Certain >99% Almost Certain ≈95% Highly Likely >80% Probable >60% Likely >51% Toss-up 45-55% Unlikely <49% Possible 1-35% Highly Unlikely <15% Almost impossible ≈5% Impossible <1% Absolutely impossible 0%

Filling out this table makes me self-conscious of the inadequacies of my current usage. But it’s still a big step forward in the War for Clarity. Indeed, reading this table might make you realize you and I suffer from illusory disagreement – or illusory agreement. And even if publishers are loathe to add a Probability Glossary, there’s no reason why every thinker who wants to signal seriousness couldn’t publicly post one on his blog or webpage.

When you do, please credit my son for the idea…