Ben Goldacre, The Guardian, Saturday 6 November 2010

Popular science is often triumphalist, presenting research as a set of completed answers, when in reality much of what gets published makes a glorious, necessary mess.

Here is an example. Solomon Asch’s legendary studies from the 1950s on conformity are among my favourite experiments of all time: some people in a room are asked to judge the length of a line; all but one are stooges, and they unanimously assert what is obviously an incorrect answer; but the one true unsuspecting experimental subject conforms to the majority view, despite knowing that it’s incorrect, about a third of the time.

It’s a chilling result that feels right, and over the past half a century researchers have replicated the study over 100 times in 17 countries, allowing hints of patterns to be spotted in the results. One analysis of US studies found that conformity has declined since the 1950s. Another found that “collectivist” countries tend to show higher levels of conformity than individualist ones.

This month the International Journal of Psychology published a new variant. Instead of one real subject in a room full of stranger stooges, they used polarising glasses – the same technology used to present a different image to the left and right eye for 3D films – to show participants different images on the same screen, at the same time, in the same room. This meant that friends could disagree, legitimately, and so exert social pressure, but without faking it.

The results were problematic. Overall, sometimes the minority people did conform to peer pressure, giving incorrect answers. But when the results were broken down, women did conform, a third of the time, but men did not. This poses a problem. Why were the results of this study different to the original study?

It could be that the subjects were different. The Asch experiments were only conducted in men, and they did conform. Perhaps modern Japanese undergraduates are different to 1950s US undergraduates (although cultural and generational differences have not previously been shown to be so large that they abolish the conformity effect completely).

It could be that the task, where you have to judge the length of a line, was slightly different. But if anything, the task in the new experiment was harder than the original, because the polarising glasses required that extra visual noise be added in, and if judgements were trickier, and therefore closer calls, then you might expect that conformity would increase, rather than decrease.

Or it could that the relationships were different. Perhaps conforming effects are less pronounced among people who know each other, rather than in a room full of stranger stooges: perhaps you feel more comfortable disagreeing with friends. This would be an important answer, if true, because when we extrapolate from the lab to the everyday, we’re probably more interested in conformity effects among acquaintances, because that’s what happens in a real community.

Maybe these questions will be resolved with a new experiment – you could probably design one yourself that would discriminate between the different possible explanations – but that will depend on whether someone is interested enough, and whether they can get the money and the time. Perhaps the paper will sink like a stone, and be ignored or overlooked, as sometimes happens with uncomfortable data.

But what you should know is this: alongside the triumphalism, and the answers, in reality, grey and conflicting results like these run deep in the research literature. They’re not an aberration, or a disappointment, in fact they are arguably the glorious norm, in the noise of over 20,000 academic journals, publishing well over a million articles every year. Alongside the giants, and the clean easy answers, challenging and ambiguous findings like these are what science is really made of.