Academic publishers argue they add value to manuscripts by coordinating the peer-review process and editing manuscripts — but a new preliminary study suggests otherwise.

The study — which is yet to be peer reviewed — found that papers published in traditional journals don’t change much from their preprint versions, suggesting publishers aren’t having as much of an influence as they claim. However, two experts who reviewed the paper for us said they have some doubts about the methods, as it uses “crude” metrics to compare preprints to final manuscripts, and some preprints get updated over time to include changes from peer-reviewers and the journal.

The paper, posted recently on ArXiv, compared the text in over 12,000 preprint papers published on ArXiv from February 2015 to their corresponding papers published in journals after peer review.

The authors report in their paper, “Comparing published scientific journal articles to their pre-print versions:”

…the text contents of the scientific papers generally changed very little from their pre-print to final published versions.

Specifically, the authors compared the differences in wording in the main bodies, titles and abstracts of preprints to the final versions of manuscripts using several text comparison algorithms.

One key limitation of the study, however, is that most of its sample consisted of physics, math and computer science papers, which are routinely posted on ArXiv before submission to a journal in the field. In a statement to us, the authors added:

We are in the process of expanding our experiments to other disciplines such as Economics and Biology. We do indeed anticipate similar results but at this stage have no hard evidence to support [this] intuition.

Originally, the sample consisted of more than a million papers, but the number soon came down — to 12,666 — after the authors discovered that many papers on ArXiv did not contain the DOIs of their final peer-reviewed versions published in journals. In the paper, the authors — all based at the University of California, Los Angeles — report further limitations of the systems used:

The main reason why this number [12,666] is fairly low is that, at the time of writing, the above mentioned CrossRef API is still in its early stages and only few publishers have agreed to making their articles available for text and data mining via the API.

(Here’s the Crossref API the authors refer to.)

When asked if the role of scholarly publishers is diminishing, the authors — led by first author Martin Klein and last author Todd Grappone — told us:

Obviously, academic publishers play a major role in academia. However, part of the argument of their added value to scholarly communications is the coordination of peer review and the enhancement of the publication’s text. Our study tried to provide empirical indicators to inform the discussion about this value statement.

But comparing differences in overall lengths, character sets or text sizes do not necessarily indicate how much the meaning of the paper — which the authors call “semantic content” — changed from preprint versions of papers to final publication.

Timothy Gowers, a mathematician at the University of Cambridge, UK, who created an ArXiv “overlay” journal Discrete Analysis (which links to papers published on ArXiv and carries out peer review) told us:

The paper uses somewhat crude metrics to measure the similarity of papers. I can imagine a change being made to a mathematics paper that would show up as a very small change with those metrics, but be of crucial importance to the paper.

He added, however:

The only significant role publishers play for mathematics is offering a “stamp of approval”, a service that can be performed far more cheaply.

Sabine Hossenfelder, a theoretical physicist at the Nordic Institute for Theoretical Physics in Stockholm, Sweden, pointed out that many authors update their ArXiv preprints with final versions of papers, so comparing the two may be a flawed method. She added:

The authors try to fix that problem by pointing out that many of the latest arxiv versions are actually posted before the publication date. This however is misleading: the authors normally know the final version that will be published when it has been accepted — that is often weeks before the paper actually gets published. Why wait with updating until the journal publishes it?

She went on:

Everybody who has ever dealt with an editor knows that even small changes to a paper can make a huge difference to it being understandable. I also don’t know why the authors want the reader to believe that changes to the length of the body of the paper are somehow not so relevant. Most of the changes during revision are that referees want explanations added.

Another limitation of the study is the authors’ lack of focus on equations, tables and figures, Hossenfelder noted:

Changing as much as an index in an equation can dramatically increase the value of a paper.

She concluded:

…if I would interpret their data, I would instead read out of this that, yes, indeed…peer review does ‘significantly’ change almost all published papers. But what does that mean for the value of the published paper?

We contacted two publishers for their views on their study’s conclusions. Rebecca Lawrence, managing director of Faculty of 1000, told us that the “key element” that publishers bring to the table is “logistical support” for the peer-review process. She noted:

What would be an interesting comparison would be to look at articles on F1000Research where the whole process of peer review and the new article versions is open, to see how much these articles were changed following peer review.

An Elsevier spokesperson added:

Academic publishers are more important than ever, continually improving the quality, discoverability and utility of an increasing quantity of published scholarly content. Publishers perform a wide range of value added activities to not only the article, but for the journal and broader community as well — some seen, some unseen.

The Elsevier spokesperson referred us to this Scholarly Kitchen article by Kent Anderson entitled “96 things publishers do,” listing the functions publishers perform besides managing peer review, along with copy-editing and formatting. In the introduction, the post notes:

Often, authors are the ones asserting that journal publishers do so little, which is understandable, as authors only experience a small part of the journal publishing process, and care about the editing and formatting bits the most, making those the most memorable. In fact, publishers’ service mentalities often include deliberately limiting the number of things authors have to worry about, which further limits their view of what it actually takes to publish a work and remain viable to publish the next one.

Like Retraction Watch? Consider making a tax-deductible contribution to support our growth. You can also follow us on Twitter, like us on Facebook, add us to your RSS reader, sign up on our homepage for an email every time there’s a new post, or subscribe to our new daily digest. Click here to review our Comments Policy. For a sneak peek at what we’re working on, click here.

Share this: Email

Facebook

Twitter

