Survey of thousands of animal studies for drugs to combat disease finds majority not rigorous enough, leading to trials that waste time, money and suffering

The search for new drugs to combat major diseases is being set back by shoddy animal research, according to work by two teams of scientists.

An Edinburgh University survey of thousands of animal studies found that the majority were not rigorous enough to rule out effects that routinely inflate the benefits of new treatments.

The analysis led by neurologist Malcolm Macleod found that top-tier journals were among the worst offenders for publishing poorly designed animal studies, and that two-thirds of animal work from leading UK universities had question marks over its validity.

Animals are used to test experimental compounds before companies decide whether to push ahead with full clinical trials in humans. But flawed animal tests can exaggerate the effects of candidate drugs, and lead to trials that end up being a colossal waste of time and money, as well as suffering.

“We could clearly be doing a lot better, and if we do better, our science will become more robust in terms of our results being translatable into new treatments for diseases,” Macleod said.



The scientists pored over animal studies to see if authors had taken basic steps to prevent biases from skewing their results. These involved the random assignment of animals to treatment groups; blinded assessment of how the animals fared; a record of animals removed from the results, and a calculation to show that the study had enough animals to be statistically sound.



Studies that fail to rule out biases can often find big effects and so get published in the highest profile journals. But these studies tend to be the most unreliable ones, and are often knocked down by better studies done later.



The Edinburgh team first looked at a random collection of 146 animal studies from 1941 to 2012 and found that less than 20% of the studies randomly assigned animals to test groups, to reduce the chances of one group, for example, being healthier than the other. Yet in only 3% of studies were scientists blinded, meaning that in 97% of cases, the authors knew which animals should show a response.



The team went on to look at more than 2,500 animal studies published in leading journals between 1992 and 2012 to see if top-tier publications reported more robust research than lower ranked journals. Again they found the opposite to be true, with less attempt to rule out biases in the top flight journals, according to their report in the journal Plos Biology.

Finally, Macleod’s team looked at more than 1,000 papers from the UK’s top universities: Cambridge, Edinburgh, Imperial, Oxford and University College London to see if the best institutions deserved their reputations. More than two-thirds of the studies did not report even one of the four measures seen as critical for reducing the risk of bias. Only one study, written by Prof Alastair Buchan, dean of medicine at Oxford, included all four.



The problem with animal research is highlighted by a second team from McGill University in Montreal which found that flawed animal studies over-estimated the effectiveness of a new kidney cancer drug by up to 45%.

The drug, sunitinib, is sold under the name Sutent and targets pathways that cause certain cancers to grow. According to Jonathan Kimmelman at McGill, studies reporting little or no cancer effect had simply not been published.



Few animal studies on sunitnib were designed to avoid biases, the scientists write in the journal eLife, and most were conducted on one laboratory mouse strain that responded well to the drug: young females with compromised immune systems.



Tests involving other animal types, including mice that had spontaneously developed tumours, showed less of an effect. In many cases it was not even clear how many animals had been tested because the sample size was not reported.



“Only a fraction of drugs that show promise in animals end up proving safe and effective in humans. An important reason is because studies in animals are often not well designed, and because positive results have a higher chance of being published. They end up skewing what we think we know about the potential of a drug,” Kimmelman said.



Buchan, who has been a researcher for more than 30 years, told the Guardian: “People are motivated to get things published, to get funded, to get promoted, and there’s an awful lot of unconscious bias going on way beyond any obvious conflicts of interest. Most of what I’ve done has found negative results and it’s very hard to continue getting negative data when everyone else is getting positive data.”



“I worry about the funding agencies not being strict enough, I worry about the investigators not reporting what they need to report, the journals need to be much stricter, and the institutions have to bear some responsibility,” he said.



Buchan has spent much of his career looking for drugs that protect the brain cells against stroke damage. “We’ve been at this for 30 to 35 years,” he said. “The reality is that in neuroprotection we don’t have a drug. And if you told me that in 1980, I would have had to think twice about going into what I thought was a really key field.”

Guidelines developed in the UK and aimed at improving standards of medical research conducted on animals were published in the journal Public Library of Science Biology in 2010. The Canadian scientists called for new guidance on the design and reporting of pre-clinical cancer studies.