Ben Goldacre

The Guardian

14 March 2009

Like you, I’ve developed a sneaking respect for all the fun and interesting tricks a person can use to distort the scientific evidence, so Dr Scott S Reuben is a double scumbag: this week, in the biggest fraud case from recent medical history, he has been caught out, rather unimaginatively, just fabricating his data.

How did he get away with it?

Firstly, if you’re planning a career in scientific fraud, then medicine is an excellent place to start. Findings in complex biological systems – like “people” – are often contradictory and difficult to replicate, so you could easily advance your career and never get caught.

And fraud is not so unusual, depending on where you draw the line. In 2005 the journal Nature published an anonymous survey of 3247 scientists: 0.3% admitted they had falsified research data at some point in their careers, in acts of outright fraud; but more interestingly, 6% admitted failing to present data if it contradicted their previous research.

They are not alone. Robert Millikan, to take just one example, won a Nobel prize in 1923 after demonstrating that electricity comes in discrete units (electrons) with his oil drop experiment. Millikan was mid-career – the peak period for fraud – and fairly unknown. In his famous paper from Physical Review he said: “this is not a selected group of drops but represents all of the drops experimented on during 60 consecutive days”.

That was untrue: in the paper there were 58 droplets, but in the notebooks there are 175, annotated with phrases like “publish this beautiful one” and “agreement poor, will not work out”. Chillingly, there is a continuum between this naughtiness, and lots of apparently innocent research activity: what should you do with the outliers on the graph? When you drop something on the floor? When the run on the machine was probably contaminated?

Dr Reuben was at the other end of the scale. He simply never conducted various clinical trials he wrote about for ten years. In some cases he didn’t even pretend to get approval to conduct studies on patients, but just charged ahead and invented the results all the same.

The details haven’t come out yet – investigators have asked various academic journals to formally withdraw at least 21 studies – but fabrication is often easier to spot than selective editing, and some people have argued for various fraud detection tools to be used more commonly by academic journals.

The human brain is a fairly bad random number generator, for example, and simple frauds have often been uncovered by forensic statisticians looking at last digit frequency: if you’re pencilling numbers into a column at random, you might have a bit of an unconscious preference for the number 7. And a more interesting version of this pattern spotting applies to the lead digits further to the left in a number, which should conform to the Benford Distribution: a mathematical formalisation of the common sense observation that 1 is more common than 9 in these positions, if you’re measuring stuff.

Fine, you might say: I’ll use a random number generator. But here you run into the problem of telltale uniformity in your randomness. Jan Hendrik Schön co-authored roughly one paper every week in 2001, but his results were too accurate. Eventually someone noticed that two studies had the same amount of “noise”, and it turned out that many of his experimental results had been generated by computer, using the very equations they were supposed to be checking, with supposedly realistic random variation built into the model.

But for all our joy at mischief, we should remember that fraud has consequences. Faking the coin can retard progress, and it can waste the time of big thinkers. Arthur Smith Woodward, one of the 20th century’s greatest paleontologists, burned valuable life at Piltdown every year until he died, trying to find more remains to match the fraudulent Piltdown Man.

And in medicine, data isn’t an arbitrary or abstract thing: Reuben’s work examined the best way to manage pain after operations, and he provided evidence that non-opiate medications are equally effective. Now that field is in turmoil. And pain really matters.

References:

www.nytimes.com/2009/03/11/health/research/11pain.html?_r=2

www.anesthesiologynews.com/index.asp?ses=ogst§ion_id=3&show=dept&article_id=12634

www3.interscience.wiley.com/journal/118500359/abstract?CRETRY=1&SRETRY=0