In one study, all the participants were presented with a numerical summary, drawn from a panel of experts convened by the University of Chicago, of the range of expert opinion on certain economic issues. On some, a large majority of experts agreed; on others, there was more disagreement. For instance, a large majority agreed that a carbon tax would be an efficient means of reducing carbon-dioxide emissions (93 experts agreed, five were uncertain and only two disagreed), but there was more disagreement about whether increasing the minimum wage would make it harder for low-skilled workers to find employment (38 agreed, 27 were uncertain and 36 disagreed).

One group of participants, however, was presented not only with the numerical summary of expert opinion but also with an excerpted comment from one expert on either side of an issue. On the carbon tax issue, for example, these participants read a comment from one of the 93 experts who thought the tax would be effective, justifying that opinion, and a comparable comment from one of the two experts who disagreed.

Then, all the participants were asked to rate their perception of the extent to which the experts agreed with one another on each issue. Even though both had a precise count of the number of experts on either side, the participants who also read the comments of the opposing experts gave ratings that did not distinguish as sharply between the high-consensus and the low-consensus issues. In other words, being exposed to the conflicting comments made it more difficult for participants to distinguish the issues most experts agreed on (such as carbon tax) from those for which there was substantial disagreement (such as minimum wage).

This distorting influence affected not only the participants’ perception of the degree of consensus, but also their judgments of whether there was sufficient consensus to use it to guide public policy. (My other study, which used a numerical summary of the views of professional film critics on various movies, as well as comments from opposing experts, had similar findings.)

What explains this cognitive glitch? One possibility is that when we are presented with comments from experts on either side of an issue, we produce a mental representation of the disagreement that takes the form of one person on either side, which somehow contaminates our impression of the distribution of opinions in the larger population of experts. Another possibility is that we may just have difficulty discounting the weight of a plausible argument, even when we know it comes from an expert whose opinion is held by only a small fraction of his or her peers. It’s also possible that the mere presence of conflict (in the form of contradictory expert comments) triggers a general sense of uncertainty in our minds, which in turn colors our perceptions of the accuracy of current expert understanding of an issue.