Philip Tetlock is one of my favorite social scientists. I often joke that every cable news show should be forced to display a disclaimer, streaming in a loop at the bottom of the screen. The disclaimer would read: "These talking heads have been scientifically proven to not know what they are talking about. Their blather is for entertainment purposes only." The viewer would then be referred to Tetlock's most famous research project, which began in 1984. At the time, the cold war was flaring up again⎯Reagan was talking tough to the “Evil Empire”⎯and political pundits were sharply divided on the wisdom of American foreign policy. The “doves” thought Reagan was needlessly antagonizing the Soviets, while the “hawks” were convinced that the USSR needed to be aggressively contained. Tetlock was curious which group of pundits would turn out to be right, and so he began monitoring their predictions.

A few years later, after Reagan left office, Tetlock revisited the opinions of the pundits. His conclusion was sobering: everyone was wrong. The doves assumed that Reagan’s bellicose stance would exacerbate Cold War tensions. They predicted a breakdown in diplomacy, as the USSR hardened its geopolitical stance. The reality, of course, was that the exact opposite happened. By 1985, Mikhail Gorbachev was in power. The Soviet Union began implementing a stunning series of internal reforms. The “evil empire” was undergoing glasnost.

But the hawks didn’t do much better. Even after Gorbachev began the liberalizing process, hawks tended to disparage changes to the Soviet system. They said the evil empire was still evil; Gorbachev was just a tool of the Politurbo. Hawks couldn’t imagine that a sincere reformer might actually emerge from a totalitarian state.

The dismal performance of these pundits inspired Tetlock to turn his small case study into an epic experimental project. He picked a few hundred political experts - people who made their living “commenting or offering advice on political and economic trends” - and began asking them to make predictions about future events. He had a long list of pertinent questions. Would George Bush be re-elected? Would there be a peaceful end to apartheid in South Africa? Would Quebec secede from Canada? Would the dot-com bubble burst? In each case, the pundits were asked to rate the probability of several possible outcomes. Tetlock then interrogated the pundits about their thought process, so that he could better understand how they made up their minds.

After Tetlock tallied up the data, the predictive failures of the pundits became obvious. Although they were paid for their keen insights into world affairs, they often performed worse than random chance. Most of Tetlock’s questions had three possible answers; the pundits, on average, selected the right answer less than 33 percent of the time. In other words, a dart-throwing chimp would have beaten the vast majority of professionals. These results are summarized in his excellent Expert Political Judgment.

Tetlock is currently embarking on an even more ambitious project. He was kind enough to answer a few questions about experts, hedgehogs and his future research.

Lehrer: You've been observing real world political experts for the last several decades. Could you tell me about some of the different styles you've observed among these experts?

Tetlock: My 2005 book, Expert Political Judgment, summarized a 20-year program of research in which we scored the accuracy of experts on a wide range of political and economic variables. We also explored experts' styles of thinking–and we found striking variation across individuals.

Some experts displayed a top-down style of reasoning: politics as a deductive art. They started with a big-idea premise about human nature, society, or economics and applied it to the specifics of the case. They tended to reach more confident conclusions about the future. And the positions they reached were easier to classify ideologically: that is the Keynesian prediction and that is the free-market fundamentalist prediction and that is the worst-case environmentalist prediction and that is the best case technology-driven growth prediction etc. Other experts displayed a bottom-up style of reasoning: politics as a much messier inductive art. They reached less confident conclusions and they are more likely to draw on a seemingly contradictory mix of ideas in reaching those conclusions (sometimes from the left, sometimes from the right).

We called the big-idea experts "hedgehogs" (they know one big thing) and the more eclectic experts "foxes" (they know many, not so big things).

Lehrer: Do these different styles correlate with levels of accuracy?

Tetlock: In assessing accuracy, it is crucial to make the "law of large numbers" work for you. Any fool can be lucky a few times. The key is consistency. So, in the first round of our studies, we assessed the accuracy of almost 30,000 predictions from almost 300 experts. We tested a lot of different hypotheses about the correlates of consistency and accuracy. Is ideology the key factor? Having a PhD? Having past access to classified information? And a lot of hypotheses bit the dust. The most consistent predictor of consistently more accurate forecasts was "style of reasoning": experts with the more eclectic, self-critical, and modest cognitive styles tended to outperform the big-idea people (foxes tended to outperform hedgehogs).

Lehrer: Your latest project aims to expand on this earlier research. Could you describe the project?

Tetlock: The current project is supported by the Intelligence Advanced Research Projects Agency of the US government – and it is the most systematic effort, to date, at testing the effectiveness of competing approaches to improving forecasting accuracy in the domain of politics and national security. Multiple teams are participating – and we are recruiting people who, ideally, have three characteristics: a deep interest in thinking about how they think (and correcting blind spots and errors); a deep interest in world affairs; a willingness to translate those deep interests into hard work (it will take time and energy to master some of the tools that we will be giving various subgroups of forecasters).

Lehrer: One of the things you've long been interested in is whether or not it's possible to improve the accuracy of expert predictions. (Given that our policy debates are heavily influenced by these experts, it's a rather pressing question.) Do you have a sense of whether or not improvement is possible? Or are we destined to always have experts that underperform random chance, if only because reality is so unpredictable?

Tetlock: I think it is useful to distinguish between radical and moderate skeptics on the question of how much room there is for improvement. Radical skeptics believe that there is zero predictability in the world – and that there is no such thing as good forecasting judgment (good judgment essentially equals good luck; that judgment equals bad luck).

Warren Buffets are flukes: if you toss enough coins enough times, a few of them are bound to land "heads" 10 or 15 times in a row. Similarly, if you have enough forecasters making enough judgment calls, a few of them are bound to look spectacularly prescient.

As a moderate skeptic, I don't go that far. I don't rule out the possibility that there are individuals who can consistently outperform chance or simple extrapolation algorithms over long stretches of time (and these individuals are more likely to be foxes and hedgehogs). But these individuals are still hard to spot before the fact. And even these individuals often experience regression toward the mean in their performance (which means that their current performance is often already deteriorating toward "just average" by the time their previously superior performance comes to public attention).

Balancing these arguments, my current gut instinct is that there is room for improvement but it may often be fairly small. That said, even fairly small improvements may be enormously valuable to society. When you're talking about multi-trillion dollar decisions, you don't have to improve the accuracy of probabilistic forecasts by much to justify multi-million-dollar investments. A 10 or 20% improvement in accuracy could quite quickly translate into savings of many billions of dollars.

Lehrer: Can non-experts do anything to encourage a more effective punditocracy? Should I feel bad about watching Meet the Press?

Tetlock: Yes, non-experts can encourage more accountability in the punditocracy. Pundits are remarkably skillful at appearing to go out on a limb in their claims about the future, without actually going out on one. For instance, they often "predict" continued instability and turmoil in the Middle East (predicting the present) but they virtually never get around to telling you exactly what would have to happen to disconfirm their expectations. They are essentially impossible to pin down.

If pundits felt that their public credibility hinged on participating in level playing field forecasting exercises in which they must pit their wits against an extremely difficult-to-predict world, I suspect they would be learn, quite quickly, to be more flexible and foxlike in their policy pronouncements.

Lehrer: What sort of subjects are you looking for in this study? And how can qualified people volunteer?

Tetlock: As I mentioned earlier, we are recruiting people who, ideally, have three characteristics: a deep interest in thinking about how they think (and correcting blind spots and errors); a deep interest in world affairs; a willingness to translate those deep interests into hard work (it will take time and energy to master some of the tools that we will be giving various subgroups of forecasters). I estimate that the total amount of work over the course of the year will be about 10 hours (we can pay a token honorary of $150 for participating for the entire year). We were required also to set a minimum educational requirement for participating (at least a baccalaureate degree – which is the minimum educational requirement for intelligence analysts).

The registration website is www.goodjudgment.info