Multi-Stage Open Peer Review (Pöschl, 2012)

One of the supposed conflicts in academic publishing is ensuring quality reviewed research in an environment of rapid scientific exchange. Traditional peer review, for instance, is a prime example of scientific quality: it allows for the dissemination of knowledge to pass through a filter of peers that self-regulates the suitability of a paper for publication. Despite not being perfect, it is currently accepted as the best system we’ve got, but one major stumbling block is the speed of publication – for anyone who has submitted a paper, the peer review process can be excruciatingly slow (to the point of competing with glacial displacement).

With that in mind, a recent paper by Pöschl (2012) argues we can have the best of both worlds: open access can provide the speed of publication we need and also improve upon scientific communication, quality assurance, evaluation and maintain sufficiently high standards of peer review:

(1) Open Access is fully compatible with traditional peer review, and in addition it enables interactive and transparent forms of review and discussion open to all interested members of the scientific community and the public (open peer review). (2) Open access gives reviewers more information to work with, i.e., it provides unlimited access to relevant publications across different scientific disciplines and communities (interdisciplinary scientific discussion and quality assurance). (3) Open access facilitates the development and implementation of new metrics for the impact and quality of scientific publications (combination of citation, download/usage, commenting, and ranking by various groups of readers and users, respectively. (4) Open access helps to overcome the obsolete monopoly/oligopoly structures of scientific publishing and statistical analysis of publication contents and citations/references, which are limiting the opportunities for innovation in scientific publishing and evaluation.

Central to Pöschl’s argument is the discussion of multi-stage open peer review: a two-stage process of open access publishing combined with iterated steps of peer review and interactive public discussion (see figure 1).

Like any successful product, multi-stage open peer review combines what worked previously with new, innovative practices afforded by open access and digitisation. Pöschl makes a convincing case for why we should adopt this approach more widely across academia and the sciences. In particular, what makes it more successful than other forms of closed or open peer review is that it enables the following:

(1) Publication of discussion papers before full peer review and revision: free speech, rapid publication, and public accountability of authors for their original manuscript foster innovation and deter careless submissions. (2) Integration of public peer review and interactive discussion prior to final publication: attract more comments than post-peer review commenting, enhance efficiency, and transparency of quality assurance, maximize information density of final papers. (3) Optional anonymity for designated referees: enables critical comments and questions by referees who might be reluctant to risk appearing ignorant or disrespectful – especially when providing a voluntary community service in which they have little to gain for investing lots of effort and time. (4) Archiving, public accessibility, and citability of every discussion paper and interactive comment: ensure documentation of controversial scientific innovations or flaws, public recognition of commentators’ contributions, and deterrence of careless submission.

For me at least, multi-stage open peer review sounds like a sensible move in the scientific community. Pöschl leaves plenty of room for manipulation of the various stages so that they are tailored for the demands and assurances of different disciplines. Still, he does outline the essential elements and scales of evaluation, as figure 3 from the paper illustrates:

If I had anything to add, then it would be my own pet interest: to try and reduce the positive results bias in science. Multi-stage peer review could feature in this, but I would also add another stage to the process. What that stage is will have to wait for one of my future posts. For now, I’ll leave you with Pöschl’s (lengthy) abstract:

The traditional forms of scientific publishing and peer review do not live up to all demands of efficient communication and quality assurance in today’s highly diverse and rapidly evolving world of science. They need to be advanced and complemented by interactive and transparent forms of review, publication, and discussion that are open to the scientific community and to the public. The advantages of open access, public peer review, and interactive discussion can be efficiently and flexibly combined with the strengths of traditional scientific peer review. Since 2001 the benefits and viability of this approach are clearly demonstrated by the highly successful interactive open access journal Atmospheric Chemistry and Physics (ACP, www.atmos-chem-phys.net) and a growing number of sister journals launched and operated by the European Geosciences Union (EGU, www.egu.eu) and the open access publisher Copernicus (www.copernicus.org). The interactive open access journals are practicing an integrative multi-stage process of publication and peer review combined with interactive public discussion, which effectively resolves the dilemma between rapid scientific exchange and thorough quality assurance. Key features and achievements of this approach are: top quality and impact, efficient self-regulation and low rejection rates, high attractivity and rapid growth, low costs, and financial sustainability. In fact, ACP and the EGU interactive open access sister journals are by most if not all standards more successful than comparable scientific journals with traditional or alternative forms of peer review (editorial statistics, publication statistics, citation statistics, economic costs, and sustainability). The high efficiency and predictive validity of multi-stage open peer review have been confirmed in a series of dedicated studies by evaluation experts from the social sciences, and the same or similar concepts have recently also been adopted in other disciplines, including the life sciences and economics. Multi-stage open peer review can be flexibly adjusted to the needs and peculiarities of different scientific communities. Due to the flexibility and compatibility with traditional structures of scientific publishing and peer review, the multi-stage open peer review concept enables efficient evolution in scientific communication and quality assurance. It has the potential for swift replacement of hidden peer review as the standard of scientific quality assurance, and it provides a basis for open evaluation in science.

Reference

Pöschl U (2012). Multi-Stage Open Peer Review: Scientific Evaluation Integrating the Strengths of Traditional Peer Review with the Virtues of Transparency and Self-Regulation. Frontiers in computational neuroscience, 6 PMID: 22783183