Read Time:

Exploring research exceptionalism

What factors contribute to research exceptionalism?

Some evidence now, even if flawed, seems preferable to expending greater resources on more demanding studies whose benefits only materialize later



The view that key features of rigorous research, like randomization or the use of placebo comparators, conflict with clinicians’ care obligations



The expectation that researchers and sponsors are generally free to exercise broad discretion over the organization and design of research





Expanding on the first point, Kimmelman says: "The proverb – 'a little bit of information is dangerous' – is operative here. For example, if people perceive – on the basis of poorly run studies – that a treatment is effective for COVID-19, they may be more reluctant to volunteer for placebo-controlled trials that rigorously evaluate that efficacy." This is an issue because it can slow the process of determining if a treatment is effective.



He adds: "Many treatments that look promising in small, preliminary reports – arguably most treatments – turn out when tested rigorously to be at best ineffective and at worst unsafe." The information that research provides is a public good, on which "caregivers, health systems and policy makers rely to efficiently discharge important moral responsibilities," the authors state in the article.



They discuss five conditions of informativeness and social value that research should embody to help the fulfil such responsibilities:





1. Importance





2. Rigorous design





3. Analytical integrity





4. Trials should be reported completely, promptly and consistently with prespecified analyses





5. Feasibility



The article describes several examples of how, in Kimmelman and London's opinion, these conditions are not being fulfilled.





Clinical trials

Nonetheless, Kimmelman and London strongly advocate the importance of adhering to the rigorous process of drug design and development that is typically enforced by regulatory agencies across the globe.



This includes the proper testing of therapeutics and preventives through clinical trials, and the scientists allude to some failings in this area: "We have in mind vaccine Phase I trials that skipped proper evaluation in animal models, or clinical trials that are not adopting blinding procedures – that is, preventing patients and physicians from knowing whether they are receiving drug or control."



In the article, the scientists also stress that "sponsors, research consortia and health agencies should prioritize research approaches that test multiple interventions".



A flurry of clinical trials that are of varying scales have emerged in recent months, taking place across different regions and sometimes testing the same therapeutic. "This massive commitment concentrates resources on nearly identical clinical hypotheses, creates competition for recruitment, and neglects opportunities to test other clinical hypotheses," Kimmelman and London say.



Can we look to work smarter in the face of a pandemic, and if so, how?



The scientists nod to recent innovations in approaches to trial design: "There are a number of new trial designs that better economize on time when evaluating treatments. Rather than testing five different treatments in five different clinical trials, you can test the five treatments in one clinical trial with one comparator arm – six arms in total. This allows you to compare treatments side by side, and it gives you better control over some of the statistical issues that arise when testing a lot of hypotheses."





Preprints: Valuable or detrimental?

"Normally, we subject scientific findings to scrutiny through peer review before they are disseminated to people who might use those findings. Via pre-print server, that filtering mechanism is not operating properly, meaning that decision-makers are often accessing information that is potentially unreliable or misleading," Kimmelman comments.



Preprints remain a relatively new concept in scientific publication, and thus, discussions surrounding their value and potential issues pertaining to their use are ongoing and pertinent.





On such topic, Kimmelman expresses uncertainty. "I’ve described above a lot of misgivings. I think if we had a coordinated effort involving an initial quality check before deposition, followed by a peer review within 24 hours of deposition, I’d feel a lot better."



Interpreting a scientific report can be quite the challenge for an individual that is not a specialist in that particular research area, meaning their critical evaluation could be inadequate. Kimmelman suggests that a mechanism to make preprints scrutable for non-specialists on a fast timeline would be a plausible solution here.



Medical journals are also placed under the spotlight by Kimmelman and London, who suggest that conclusions are being drawn from data without assessing the full context of the experiment. Kimmelman told Technology Networks: "Publications in prestigious medical journals [are] claiming that patients receiving a drug through “compassionate use” benefited, even though there was no control group or study protocol. Here is a classic example of exceptionalism. Normally a top tier medical journal would not publish a paper like this; it would be more likely to use the paper it is printed on as bedding in a hamster cage."







"One lesson of the current outbreak is that expeditious research in a crisis situation is feasible," - Kimmelman.

"Getting to the finish line"



Photo credit: Steven Lelham on Unsplash.





"The most efficient way to get to the finish line is to demand exacting standards of evidence. Relaxing those standards – even if the individuals doing so have good intentions – only threatens to prolong our uncertainty and waste resources," Kimmelman concludes.



Jonathan Kimmelman was speaking to Molly Campbell, Science Writer, Technology Networks.



Reference: Alex John London and Jonathan Kimmelman. (2020). Against pandemic research exceptionalism. Science. DOI: https://science.sciencemag.org/cgi/doi/10.1126/science.abc1731.