Peer review is often claimed to be the guarantor of the trustworthiness of scientific papers, but it is a troubled process. Preprints offer a way out

A few weeks ago my collaborators and I submitted our latest paper to a scientific journal. We have been investigating how noroviruses subvert the molecular machinery of infected cells and have some interesting results. If it passes peer review, our paper could be published in three or four months’ time. If it’s rejected, we may have to re-work the manuscript before trying our luck with another journal. That will delay publication even further – it’s not unheard of for papers to take a year or more to get out of the lab and into the world, even in the digital age.

But you can read our paper today, for free, because we have uploaded it as a preprint to the bioRxiv (pronounced ‘bio-archive’). This was an unusual thing for us to do. Preprints are a relatively new thing for life scientists, though the arXiv (‘archive’) preprint server has been in use in many fields of physics, mathematics and computer science for over 20 years. To be honest, it felt odd to be publishing without the comfort blanket of peer review. We went ahead anyway because preprints are part of the solution to the troubled state of research publication and we want to see more scientists publishing by this route.

But isn’t peer review supposed to be the quality assurance mechanism for research, an essential filter that prevents flawed or nonsensical papers from being published? It is often touted as such in reassuring tones when scientists talk to the media or to the general public – especially in discussions of politically contentious areas such as climate science, vaccine safety or genetic engineering. Are delays in publishing not a price worth paying to ensure the trustworthiness of the published literature?

In science we trust… up to a point Read more

Well, yes and no. And picking apart the answers to these questions reveals a great deal about the state of scientific publishing, which is very much in flux. It also lays bare the sometimes self-contradictory motivations behind scientists’ decisions on how and where to publish their work.

Peer review is a complex task performed by busy people. It should come as no surprise that it is an imperfect process but, because peer-review reports are generally not published, its limitations are largely invisible to the public. However, they are well known to scientists.

For the most part, reviewer identities are hidden from manuscript authors. The cloak of anonymity allows reviewers to make their assessments as frankly as possible. Although many do so professionally and responsibly, reviews too often also reveal the darker human impulses at work within the research community – ego, jealousy, ignorance and laziness. Every researcher I know has at least one tale of woe about their experiences at the hands of unreasonable or incompetent peer reviewers. Michael Eisen, a biologist at the University of California Berkeley, has described peer review as “conservative, cumbersome, capricious and intrusive”. Richard Smith, a former editor of the British Medical Journal, has studied the practice more systematically than most and concluded that “it is ineffective, largely a lottery, anti-innovatory, slow, expensive, wasteful of scientific time, inefficient, easily abused, prone to bias, unable to detect fraud and irrelevant.”

Questions about the efficacy of peer review keep coming up. The most selective journals, which one might suppose to have the most rigorous peer review, are also those with the highest rates of retractions (papers deleted from the published literature due to serious errors or fraud). There is a growing concern that the results of many peer-reviewed studies cannot be reproduced – supposedly the benchmark for good science. And new troubles are emerging, exemplified recently by the scandalous sexism of a reviewer for the journal PLOS ONE and by news last month of another set of retractions triggered by the belated discovery of authors who had used fake email addresses to masquerade as reviewers of their own work.

I don’t wish to give the impression that the present state of peer review is hopeless – it’s not, though the precise scale of the problem is difficult to determine. In any case, some serious efforts are now being made to improve peer review. Moves to open up the process, by publishing reviews (and, sometimes, reviewers’ names) are being adopted by more journals and should encourage more professional behaviour. Increasing demands for authors to make the raw data associated with their papers freely available are reducing the scope for fraudsters (as well as enhancing the re-use value of published research). Separately, attempts to validate published findings through various reproducibility projects should help to root out poorly conducted work and, at the same time, raise awareness among researchers about the steps needed to avoid falling into error.

For what it’s worth, my own experiences of peer review as an author have been mostly positive and I am grateful for the insights that have improved my published work. One of the things that made uploading our latest work to the bioRxiv somewhat nerve-wracking was that there were no reviewers to catch any silly mistakes before our manuscript went public. In turn, I hope I have provided a constructive critical eye in serving as a reviewer.

But my sense is that the utility of peer review is mostly as an adjuster rather than a filter of research publications. Reviewers of my journal submissions have sometimes made constructive suggestions for additional analyses or experiments, though as often as not my group has had to battle demands for modifications of questionable value that were based on a misreading of our intent. While our papers may have been clarified and improved in the process of review, errors or inadequacies in the analysis have still got through, and only been corrected after publication by sharp-eyed readers or follow-up work from other labs.

Such experiences may be more bruising because the correction is made in public but this type of ‘post-publication peer review’ is as much a part of the research process as the pre-publication sort and, arguably, more effective. Eisen and Smith, now advocate the exclusive use of post-publication peer review as a more open and transparent means of bypassing many of the problems of pre-publication review.

I’m not yet ready to completely abandon journal-based review myself – I’d still like to see all papers pass some sort of triage before formal publication – but I feel the ground moving. The growing use of preprints is an important part of that shift because they are a reminder to scientists of the most important purpose of research publication: the rapid dissemination of new results so they can be read, critiqued and built upon. We have lost sight of that because scientific publication through journals has become more about earning prestige points to advance your career than communicating new findings. This has perverted both the motivations of authors and the job of reviewers.

Researchers compete intensely for publication in the journals with the highest prestige (evaluated crudely as an impact factor) because they know that career success is critically dependent on having a CV stuffed with papers in the ‘top’ journals. Superficially, that might not appear to be problematic: competition for prized spots in the most highly regarded journals drives scientists to do some of their best work and there is no doubt that the best journals publish plenty of outstanding research.

Who governs science? | Stephen Curry Read more

But the excess rewards for publishing in ‘top’ journals are incentives to corner-cutting (stories streamlined by the omission of inconvenient data have more appeal) and fraud, and are undoubtedly contributing to the problems of reproducibility. A more common and invidious problem is that the prestige won by publication is now far more important than sharing results rapidly: scientists readily accede to long delays as they chase after the journals with the highest impact factors, often subjecting their papers to multiple rounds of rejection and resubmission.

The job of the reviewer also distorted: it is more often now to decide, not whether a manuscript is any good – providing a clear report of a well-executed investigation – but whether it is good enough for the journal considering publication. For ‘top’ journals that can depend as much on topicality or newsworthiness as scientific quality.

All of these problems are well known but the tragedy for science is that too few are willing to break away from the present system. However, preprints may be a way out of the impasse because they don’t require a radical change of behaviour, as the eminent cell biologist Ron Vale argued recently – fittingly, in a preprint. That may seem an odd claim in view of the fact that even after 20 years of the arXiv, preprints have not been adopted universally. The slow uptake is a reflection of the inherent conservatism of scientists – the traditional paths to publication are too well worn – but probably also due to the widespread misconception that journals will not accept manuscripts that have been posted online as preprints. This is simply not true for most major publishers.

Many researchers probably also fear that publication of papers that have not been peer reviewed risks opening the floodgates to junk science, but this has not been the case in practice. The arXiv requires authors to be endorsed by an existing arXiv author before they are allowed to post preprints, while the bioRxiv has a large panel of affiliates (disclaimer: I am one) who check that uploaded manuscripts contain “biological science as an academic scientist would understand it”. In practice, both vetting systems appear to work well.

Serious authors provide their own internal vetting in any case since most are keen to protect their reputation. Even without external peer review – and I know this from my own experience of submitting to the bioRxiv – authors are likely to take great care with preprints because they are immediately opened up for critique and discussion by a worldwide community of reviewers.

The practice of providing open feedback to authors is not yet so common in the arXiv, which doesn’t have a commenting facility, but preprint authors sometimes receive helpful critiques by email. Other forms of commenting are more indirect: the arXiv is consulted daily by the communities that use it and stimulates plenty of informal discussion over coffee, in journal clubs and on bulletin boards. These conversations, wherever they happen, are seen as one of the arXiv’s greatest benefits. Molly Peeples, an astronomer at the Space Telescope Science Institute in the US, tweeted to me enthusiastically:

Molly Peeples (@astronomolly) @Stephen_Curry oh cool! I hope it picks up...the arXiv has a huge and pervasive positive impact on astronomy culture as a whole

The bioRxiv, born in the age of social media, has a built-in comment feature. This facility is still relatively under-used – about 10% of papers uploaded so far have attracted comments – but that may well be because life scientists are still growing accustomed to the new platform. A commenting level of 10% is nevertheless remarkable because it far exceeds the commenting rate on papers published in peer-reviewed journals (though Pub Peer has emerged as an important third-party service for hosting critical discussions of published work). Moreover, the tenor of most comments is constructive. Richard Sever, assistant director at Cold Spring Harbor Laboratory Press, which runs the bioRxiv, reckons this is “because commenters see a chance to affect the course of the paper”.

That is certainly the experience of Oxford psychology professor Dorothy Bishop, who recently posted a manuscript on PeerJ Preprints, another relatively new platform that has been attracting comments and questions on well over 20% of its submissions. Bishop told me in an email that her preprint stimulated a constructive and critical discussion that might not have happened through traditional channels of peer review:

“I found the whole thing very positive. Our paper was a critique of a published paper, and we got an extensive critical commentary from the original authors. A case like this is not well handled by the conventional journal system, because either you use the authors of original paper as reviewers - but then there is conflict of interest and the possibility that unscrupulous rivals might block a critical paper; or if they aren’t reviewers they may justifiably complain they were misrepresented. A preprint has three advantages; (a) original authors could have their say and we could take on board the points they raised; (b) commentary is all out in the open so readers can evaluate the quality of the arguments; and (c), if our paper is rejected we still have the preprint with comments out in the public domain, so the work isn’t wasted.

Preprint servers are enabling the informal discussion of scientific ideas on a global scale that was once confined to correspondence between pairs of individuals. Open commentary on preprints is also increasingly being incorporated into the management of peer review by innovative journals such as PeerJ, F1000 Research, Atmospheric Chemistry and Physics and, the newest kid on the block – launched just last week – Research Ideas and Outcomes.

There are other advantages too. Preprints could become an effective outlet for negative results. These are a vital aspect of the scientific process, but have for too long been extremely difficult to insert into the scholarly literature where there is an excessive preoccupation with new discoveries and the confirmation of beautiful hypotheses.

Traditional-minded scientists might be further persuaded by evidence that publication in the arXiv immensely increases the number of times papers are read and cited by other researchers. This is a potent demonstration of the efficacy of dissemination through preprint servers. It should give pause for reflection to those who insist publication in the right journal is needed to reach the right audience.

Preprints are no miraculous panacea. Our fixation with journal prestige is too deeply embedded and for that reason, they are not about to displace journals. Nor are they likely to eliminate all the imperfections of peer review – science remains too human and argumentative an enterprise. But they are a way of challenging the status quo. By harnessing the culture of openness and accessibility that has been engendered by the web, and by recalling the collaborative, amateur ethos that is still at large within the scientific community, preprints can help to refocus attention where it matters – on the work itself, not where it is published. In so doing, they have the potential to revitalize the scientific enterprise. If a late adopter such as myself can see the benefits, there has to be hope for others – all the more so because publishing work as a preprint is a small, practical step that many researchers could take today (though unfortunately I have yet to find a preprint server aimed at chemists).

Of such small steps are revolutions sometimes made.

1 This is not to say that preprint archives can be run for free but their operating costs are minuscule compared to most journals.



@Stephen_Curry is a professor of structural biology at Imperial College and is grateful to Jenny Rohn and Dorothy Bishop for comments on this preprint.



