What are the barriers to post-publication peer review?

At ScienceOpen, we have over 28 million article records all available for public, post-publication peer review (PPPR), 3 million of which are full-text Open Access. This functionality is a response to increasing calls for continuous moderation of the published research literature, a consistent questioning of the functionality of the traditional peer review model (some examples in this post), and an increasing recognition that scientific discourse does not stop at the point of publication for any research article.

In spite of this increasing demand, the uptake of PPPR across different platforms seems to be relatively low overall. So what are some of the main reasons why researchers might feel less motivated to do PPPR, and is there anything we can do to increase its usage and adoption as part of a more open research culture?

What even is ‘post-publication’ peer review?

There is a general mentality among researchers that once research has been published, it has already ‘passed’ peer review, so why should it need to be peer reviewed again?

In reality researchers spend their time ‘peer reviewing’ research by reading papers critically, re-analysing their data, checking methods, running someone else’s code, and synthesising information across articles to add new context and information. We don’t even realise it most of the time, as this is just the subconscious way in which we perform research on a daily basis. Traditional pre-publication peer review is just formalising this part of the natural research process.

One problem is that, sadly, much of this important context and evaluation remains locked away in notebooks and hard-drives, either lost to research forever, or delayed until it is published some years down the line. Which is a shame, because the value of this process goes largely unrecognised.

PPPR, then, is about making parts of this everyday research process more public. It’s about taking the evaluations and critiques that researchers, and others, perform and adding that context to published research articles.

Everyone benefits from this process – authors gain feedback and important criticism of their work; external researchers gain a deeper insight into research; and we all increase our general understanding of a topic. It’s the same as any other ‘social evaluation’ platform like Amazon – we use reviews to refine our knowledge of a ‘product’, or research paper, compare across ‘products’, and ultimately use this for advancing our own research as part of a wider community.

Barrier: PPPR takes too much time and effort

Almost everyone in research, be they Master’s students, PhD students, Postdocs, Professors, or technical engineers, are overworked, underfunded, both, or even unemployed. For this reason, it is quite understandable why many turn down extra unpaid work (i.e. “peer review”). Never mind that traditional peer review already takes up a lot of time and effort, and despite great initiatives like Publons*, still largely goes unrecognised across the board. The same still applies to PPPR too, because it’s seen as an additional workload on top of ‘normal’ peer review.

Research has shown that academics are in one of the most over-worked professions out there. There is never enough time, with research, teaching, drinking coffee, writing papers and grants, and all the other demands on researchers, to do anything that isn’t top priority.

So how do we resolve the PPPR versus time conflict?

Potential solutions

Well, peer review doesn’t have to take a long time. It can, depending on the paper, but it doesn’t have to. To do a thorough peer review can sometimes take days. But we’ve all had an experience of that peer review which took 8 months to be completed, and in the end came back as half a paragraph of text recommending the smallest of modifications, and clearly the reviewer had only spent half an hour reading the paper on before responding.

The current system of traditional peer review is more time-consuming, causes publication delays, is more costly, subjective, secretive and exclusive, and overall vastly more inefficient than any post-publication process will ever be. Every argument levied at PPPR can be said of the traditional process too, and that is especially true of the ‘time and effort’ angle.

But to make things much simpler, PPPR doesn’t always have to be a completely detailed or comprehensive analysis of all aspects of a research article. If enough people comment on different parts of a paper, that eventually will form a complete whole. We know that at least some level of quality control has already been performed in the first place to get an article published – it’s our collective job to increase that quality where we can. And not every article has to have PPPR – it just has to be injected by the right person for the right article at the right time.

A peer review doesn’t have to cover all aspects of a paper to be useful either. If you’re an expert in data analytics, comment on the data analytics section of an article. If you’re a bibliographic wizard and notice that relevant articles are missing from the context or discussion, leave a peer review or comment pointing out the issues.

Did you find a flaw in the data or code? That’s a peer review – make it public. Is someone’s method awesome, and work really well? That’s a peer review – make it public. Is someone completely ignoring an important subset of research for some reason? That’s a peer review – make it public. I think you see where this is going now.

The point is that the detail of peer review comprises a spectrum, and as a result does not always have to be time consuming or a huge effort. What is important is sharing your knowledge in a timely manner that adds positively to research discourse.

Barrier: Lack of motivation and duty

We have written before about how editorial control will always be a critical aspect of any open peer review system, including PPPR. Editors are required to solicit peer reviews in a timely fashion (and often send numerous reminder emails), as well as provide the sort of ‘prestige factor’ with being requested to review an article, as a direct acknowledgement of your expertise in that field.

Among the other activities researchers have, PPPR might therefore be seen as relatively important, but not yet necessary. With increasing questions arising as to the validity of the published research record though, it is only a matter of time until PPPR is seen as a way of facilitating post-publication moderation and correction of research articles, and becomes as commonplace as using Dropbox or GitHub to aid the overall research and communication process.

Potential solutions

Therefore PPPR could, and probably should, be viewed more by institutes, research funders, and researchers themselves as a relevant source of quality control to reflect the natural state of academic publishing – that published research is never perfect. Simply being published is no guarantee for the ‘correctness’ of research, or longevity as such. PPPR forms an essential part of the recognition that research is a continuously evolving process built upon discretised ‘discoveries’ published as research papers.

Furthermore, due to the competitive nature of academia, it is entirely possible that some researchers have a mentality like “Why should I help a competing researcher improve their research?” This is entirely consistent with a system in which mutualism or symbiosis is rewarded less than personal or individual progress, due to limited resources and a highly competitive work environment.

If the research record was perfect because it has ‘passed’ peer review, we wouldn’t need to keep doing research – indeed, it is the imperfection of peer review and the published record which motivates us to continue to do research! A good researcher never blindly cites the research without critical evaluation of it first – that’s our job. That citation is an indication that a researcher (i.e., a peer) has evaluated (i.e., reviewed) that article subsequent to its publication.

One thing that pops up all the times in discussions of peer review is that many researchers see traditional pre-publication peer review is part of their duty as researchers – not to the journals, but to their research community. The same is exactly true of PPPR too – continued evaluation of the published record is part of that same duty to your research community.

If researchers begin to perform continuous quality control as part of their responsibility, then perhaps we will see an increase in trust levels in published research again, a factor which is all too critical in the current global social and political environments.

Barrier: Too many alternative methods or platforms of communication

PubMed Commons, ResearchGate, PaperHive, and PubPeer are just several examples of platforms or tools that allow you to perform some form of post-publication commentary on research articles. When multiple copies of papers exist across multiple platforms, it is possible to have different comments appearing on different sites but not others. It’s also possible for researchers to suffer from ‘platform fatigue’ – too many competing websites trying to achieve the same thing, and fragmenting the landscape by doing so.

Additionally, if someone has a genuine critique of an article, they might wish to make it ‘formal’ by publishing a response commentary, and get the recognition that comes with that as an additional publication. This mode of communication is probably more appropriate though only when serious theoretical or empirical flaws have been found in published research.

The problems here stem from a lack of comprehensiveness, interoperability, and critical mass uptake as the de facto platform for PPPR. The result of this is a mess of different platforms having different types of commentary on different articles, or sometimes the same ones, none of which can be viewed easily in a single, standardised way. That doesn’t seem very efficient.

Potential solutions

We thought pretty deeply about this issue at ScienceOpen too, and this is why we’ve made our platform as integrated as possible with other parts of the scholarly communications infrastructure. We have partnered with other key players such as Crossref and ORCID to make sure that when you perform a PPPR on our platform, it doesn’t conflict with your other profiles, but works with them instead.

Each PPPR at ScienceOpen becomes a ‘publication’ in itself, moderated by ORCID, accredited by Crossref by assigning each one a DOI. We attach a CC BY license too just to make sure each is viewed as a completely open publication, easily citable and can even be integrated further with your Publons and ORCID profiles. As far as alternative platforms for PPPR go, that’s about as integrated, inter-operable, and accredited as you can get!

Barrier: Appearing confrontational in public, biases, and abuse of power dynamics

Some researchers, especially those in more junior positions, are afraid that if they are vocal or negatively critical about another researcher’s work in public, especially one who is more senior, then there could be a potential backlash against them for it. Documenting this sort of power dynamic abuse is difficult to see or measure, but certainly impacts upon earlier career researchers, who are perhaps those more willing to engage in more open research practices. At the moment, it is a shame that this sort of behaviour exists within research environments, and rather than being addressed at source is being used to constrain different parts of ‘open ecosystems’. Rather than using bad behaviour as a reason for remaining ‘closed’, we should be asking why we don’t address poor behaviour as a systemic issue where it exists, especially during traditional peer review processes.

One major issue with the current system of peer review is that it empowers bad actors with position and status to marginalize those with relatively less power. This means that the inverse is also possible too, that some people might use PPPR to be deliberately confrontational in public, and to talk down to or intimidate their junior peers. Therefore, any alternative or complimentary system has to reduce or minimalise this negative dynamic, make sure that an accountability process is inbuilt and managed, and that marginalised communities feel invited to participate. Too many social platforms already have comments that are negatively geared against women or minorities, and therefore any PPPR platform has to have a mechanism in place to reduce or negate this entirely.

Potential solutions

At the moment, there doesn’t seem to be any consensus on how to resolve this issue. There are arguments for increasing anonymity (blinding peer review) for protection, and also increasing openness to expose bad behaviour and make researchers more accountable. These arguments apply to both pre- and post-publication peer review, and at the moment there is generally insufficient data on the impact of anonymity on PPPR.

At ScienceOpen, we have found that all PPPRs performed to date on our platform have been civil and constructive. All reviews have been named, as this is a pre-requirement due to the integration with ORCID. This doesn’t mean that we have managed to eliminate all potential biases and issues, but it at least seems to be working better than the traditional system, in which the many biases still largely remain (see eg a recent study on gender bias here).

We recently had a case were groups of students tested the reproducibility of a paper on our platform. Each post-publication peer review was courteous, detailed, and constructive. The author of the paper responded that they were delighted with the attempts to critique and improve their research, with no ill side effects to those performing the evaluations. This is how PPPR should be: as a progressive, open, and above all civil and professional dialogue.

Barrier: No-one reads or uses PPPRs

Further questions arise as to the actual readership of PPPR comments. What if substantial issues are raised, and the authors just ignore them? Are they going to go back and address comments on research that might be years old, and funding has completely run out on? Commitment to perform a PPPR on an article might be difficult if a guaranteed reciprocal commitment to address any issues raised is not given. One problem here, as pointed out by others like Lenny Teytelman before, is that a lack of version control over the vast majority of the research literature makes actually ‘adapting’ papers to include post-publication comments is impossible. This is because, despite the functionality that the Web provides us, the vast majority of research articles are still published as static ‘papers’, with single versions that are considered final and uneditable. What we have then is not evidence of a problem with uptake of or demand for PPPR, in this case, but evidence for lack of an incentive to do PPPR as there are very little real consequences of doing so.

Is anyone even going to read PPPR reports, or are they just going to gather dust as footnotes? How many researchers even have heard of PPPR, or know what it is or what it is for? So, the question then becomes are PPPRs really even that useful, academically? Why should anyone spend their time trying to improve a research paper if the authors won’t or can’t actually then improve it? PPPR therefore becomes a communications issue, based around cultural norms and practices within academia.

Possible solutions

Possible ways to resolve this include trying to maximise PPPR reports’ visibility and re-usability. At ScienceOpen, we are similar to other journals like F1000 Research, in that PPPRs are presented on the same pages as the articles themselves. Reviews are clearly presented with summary statistics, names, graphs, data tables, and DOIs to make them as visible as the research articles themselves.

Another solution is for publishers to start using version control with peer review, and provide updated versions of papers with successive rounds of peer review. This is what we do at ScienceOpen Research, and also at other journals such as F1000 Research – my personal experience is that this is a far superior method of publishing than any traditional model. However, for now, the vast majority of the research literature requires overlay (e.g., PubPeer) or aggregation (e.g., ScienceOpen) features to directly link PPPR to papers so that readers can easily see them, until the immense value of version control is recognised and it becomes more widespread.

How can we make PPPR on ScienceOpen more attractive?

We see post-publication peer review as a way of making part of the standard research process more useful for everyone, providing additional context to published research, improving the overall research process, and demonstrating the knowledge of individuals in an impactful way. What we facilitate is the publishing of critical post-publication evaluation of research at any level. At the present, this can be done for almost 30 million article records on our platform, and this number is growing ever day!

The important thing is attaching post-publication evaluation to the articles themselves. Blogs and other social commentary are almost valueless and lost into the void if not linked to articles. It needs to be a permanent record to be of value to future research. Next time you’re commenting on a paper on a blog, Facebook, or Twitter, consider posting it as a PPPR instead and making it a permanent and more valuable contribution to the research record.

We showed recently the immense power of publishing replication analyses as post-publication peer reviews on our platform, both as an analytical training mechanism and as a way to critically progress research. We even upgraded our PPPR features so now users can:

Add images and tables to reviews

Register reviews in CrossRef through DOIs

Add reviews to Publons and ORCID profiles

But what else can we do to encourage the uptake of PPPR? For example, what if we allowed everyone to comment and rate articles, but just like with blog post comments we should allow authors to decide on whether they make publicly visible those feedback or not. Or would it completely go against the institution of PPPR? What about changing the ORCID moderation of who can perform a PPPR (users currently need a minimum of 5 records attached to their ORCID) – is that fair? How can we make PPPR on ScienceOpen more inclusive so that everyone feels like that they can contribute equally? How can we make it so that reviews are even more recognisable?

Any thoughts, queries, questions or comments on this topic would be greatly appreciated from the community! In the mean time, we hope you take try PPPR-ing an article on ScienceOpen, and help contribute to research in the open.

Author note: Thanks to everyone who contributed to the Twitter discussion on this topic recently. At ScienceOpen, we greatly value the input of the research community, and always listen to how we can improve our services for you.