Susan Lorns

Scientists rarely get credit for one of their most important jobs — the unpaid task of peer reviewing papers. The startup Publons was created to change that by encouraging researchers to post their peer-review history online. Funded with about $US300,000 from angel investors in New Zealand, the company also rewards top reviewers with prizes, such as credits for Amazon Web Services.

On 8 October, Yogendra Kumar Mishra, who studies nanomaterials at the University of Kiel in Germany, was crowned as Publons’ top reviewer for the past three months, on the basis of the volume of reviews he recorded on the site, plus merit points earned for achievements such as reviews being endorsed by others. Nature spoke to Mishra; Malcolm Jobling, a fish biologist at the University of Tromsø in Norway, who won a previous Publons award and has racked up more reviews than any other user; and Publons co-founder Andrew Preston, a former academic based in Wellington.

What is the purpose of Publons and how does it work?

Preston: Peer review is one of the pillars of research, but reviewers don't receive recognition for reviewing. And, with a few exceptions, their pre-publication reviews are discarded as soon as articles are published. That's a lot of valuable context and insight going to waste. We want to encourage sharing of peer reviews and discussion. The first step, we think, is to turn peer review into a measurable research output.

Scientists can sign up to Publons and record their history of peer reviews, both before and after publication. The text of pre-publication reviews is not revealed unless the journal allows, but we ask for the review receipt, or we work behind the scenes with a journal editor, to verify that it took place. We are working with journals to streamline this process so that one’s review record can be auto-updated. If a journal is tied to anonymous review then our platform can still be used to give a reviewer verified credit.

Why would a scientist publicly list their peer-review record?

Preston: When you think about it, being selected to peer review a paper shows that the editor considers you an expert in your field. Publons is a great way to highlight that expertise. Our users are starting to use their records on their CVs and also to apply for things like editorial boards.

Jobling: Peer-review is the unrecognized cornerstone of publication — without evaluation and quality control, the entire system would fall apart. If Publons can serve to highlight the important role of reviewers and make the role more visible to both the academic community and, not least, administrators, then it will have done us all a great service. I already post my editorial board memberships and reviewing activities on Norway’s Current Research Information System, CRIStin. Publons may serve to give this type of registration more credibility, but it will probably take a very long time before full recognition is achieved.

How many reviews do Publons' top reviewers register each month?

Jobling: I have reviewed more than 125 manuscripts this year already, and on Publons registered 39 in the past three months. Generally I review about 100 manuscripts per year. I am sure that this is far more than most others.

Mishra: I review around five papers per month, and on Publons registered 22 in the past three months. I have no clue about other scientists, but think most of them will be in a similar range, around three to five papers per month.

How long do you spend on each review, and how many requests do you turn down?

Jobling: It varies widely. I may review a poor manuscript in about three hours; for a manuscript with some merits but requiring major revisions, I could spend 12 hours. I turn down about three times as many reviews as I accept. There is certainly an increasing number of requests. I sometimes feel overwhelmed if four or five requests come in during the course of a single day — not an infrequent occurrence.

Mishra: It entirely depends on the level of the paper and the journal to which it has been submitted. Some of my colleagues say they don’t review papers from low-impact-factor journals, but I do. The number of review invitations has increased dramatically. For me the subject matter of the paper is the main interest. I turn down papers I am not interested in.

Trine Holm Larsen/Malcolm Jobling

Should pre-publication peer reviews be made public? And should reviewers be anonymous?

Jobling: I would support making reviews public once a manuscript has been published, provided the journal, authors and reviewers had agreed. Ideally, in such a case the whole process should be transparent, from the first submission of the manuscript through initial review, revision, re-review and so on. At present I keep my reviews anonymous — I have done some signed reviews but on such occasions have sometimes been contacted by authors who have not only expected additional help but have seemed to want me to rewrite the manuscript for them. Based on these experiences I have now opted to remain anonymous.

Mishra: In my opinion, reviews should definitely not be made public. I can see some people have done it on Publons — but I will try not to do this.

Preston: Almost all the peer reviews we record are not open — so we can only record a credit, rather than the review text. The journals PeerJ and GigaScience do open reviews and automatically import the text to Publons, and we index other open-review sites like F1000. In the long run I think we'll start to see more and more open review in general, and on Publons for sure. Our merit system does award more points to open reviews but we recognize that blind review plays an important role in science. No matter what type of review it is, our goal is to build a positive culture where reviewer contributions are recognized and peer review is something that authors, reviewers, and editors all benefit from.

And the way I see the post-publication review landscape developing, there has to be an incentive to review or comment before people will start to do it in bulk. We believe this will involve creating a cross-publisher profile, so no matter where you comment, you get recognition at your central profile.