An autism researcher and an Oxford professor have exposed what appears to be a substantial case of academic malpractice, highlighting the vital role of peer review and the dangers faced by whistle-blowers

One of the most frustrating things we see as researchers is the glacial pace at which attitudes change in academic science. A culture of hidden peer review, hidden data, paywalled journal articles and performance-related bean counting undermine transparency and robustness in science. In some cases, gaming of research practices can reach the point where it threatens the integrity on which science so crucially depends. Unfortunately, the people who shine the light on such behaviour often become the target of baseless criticism and attack.

Dorothy Bishop, Professor of Developmental Neuropsychology at Oxford University and Fellow of the Royal Society, is no stranger to this sort of criticism. Last month, following concerns raised by autism researcher Michelle Dawson, Bishop started investigating editorial practices at two journals - Research in Developmental Disabilities (RIDD) and Research in Autism Spectrum Disorders (RASD) – and the relationship between these journals and founding Editor-In-Chief, Professor John Matson of Louisiana State University. Her findings are extraordinary and the criticisms she has faced as a consequence of going public are deeply disheartening.

When self-citation becomes a problem

Matson has a spectacular publishing record – around 800 papers in a career spanning over 30 years. Until recently, his university web page listed him as ‘highly cited’ by Thomson Reuters (this graphic has since been removed), and he has a h-index of 59, which means that at least 59 of his papers have been cited at least 59 times. However, as Bishop has found, part of the reason that they are highly cited seems to be because Matson has an inordinately high self-citation rate. For example, this paper published in March 2014 cites 64 papers, and 36 of them list Matson as a co-author.

Self-citation isn’t an unusual or necessarily dubious practice – it can often make sense to refer to your own previous work when discussing research in the same area. But to be self-referential to such an extent implies a gaming of the system to boost one’s own citation metrics (since most measures, like the h-index, do not distinguish between self and non-self citations). In Bishop’s analysis, Matson’s self-citation rate is about 55%, compared to an average of about 5% for other researchers in the same field. We asked Matson why his self-citation rate is so high, but he did not respond.

Peeling the onion of a flawed peer review process

The problems uncovered by Bishop, however, seem to go much deeper than self-citation rates, and include researchers other than Matson. In a follow-up blogpost on 24th February, Bishop produced data showing that for 32 papers co-authored by Matson from 2010-2014 in another journal, Developmental Neurorehabilitation, the median time between the article being submitted and being accepted was 1 day. To put that in context, the Royal Society provides information on their average times to a decision being made on a manuscript for five of their journals here, and the range is between 23 and 43 days. That’s not to say that sometimes researchers don’t experience super-fast turnaround times for manuscripts, but 1 day seems to be pushing it to the extreme.

Matson isn’t the only academic to benefit from what might be generously referred to as an “extremely efficient” review process. Bishop’s analysis also identified other researchers who have published frequently in RIDD and RASD, including Jeff Sigafoos, Mark O’Reilly and Giuliano Lancioni. Bishop has provided data showing that for 73 papers appearing in RASD and RIDD co-authored by these researchers between 2010 and 2014, 17 were accepted the same day that they were received, 13 within one day, and 13 within two days. We contacted Sigafoos and Lancioni with this data, and they responded:

The figures you state for 73 papers is routine practice for papers published in RIDD and RASD. A large percentage of all papers published in any given issue of RIDD and RASD appear to have received a rapid rate of review as indicated would happen in the official editorial policy of these journals.

In other words, they believe they were acting within the scope of the editorial policies for these journals. The fact is, they were, but these editorial policies are bizarre. Here is the official peer review policy for RIDD:

In order to maintain a rapid rate of review all submitted manuscripts are initially reviewed by the Editor in Chief for completeness and appropriateness to the journal’s stated Aims and Scope. Manuscripts that pass the initial review will be handled by the Editor, sent out to reviewers in the field, sent to an associate editor for handling, or some combination thereof, solely at the discretion of the Editor.

The first sentence in this policy makes sense. Journal Editors-in-Chief often take on a ‘triage’ role, which involves rejecting a lot of submitted manuscripts without sending them out to peer review. This is usually because the manuscript isn’t appropriate for the journal, or generally isn’t of a sufficient quality by whatever measures are appropriate for the journal in question. However, the second sentence of the policy is extraordinary, and appears to give the Editor-In-Chief carte blanche to accept manuscripts without peer review. Manuscripts will be handled by the Editor, sent out to reviews, given to an associate editor, or some combination thereof.

RIDD and RASD therefore adopt a policy in which the Editor-In-Chief can, according to the rule of the letter, bypass the peer review process and accept manuscripts outright. This would go some way to explaining the amazingly fast turnaround times on some of these articles, but it is inconsistent with the widely agreed ethics of peer review. The Committee on Publication Ethics (of which RIDD and RASD’s publisher, Elsevier, is a member) has clear guidelines stating that all journal content, apart from editorial material, should be subject to peer review from outside experts.

When is a developmental disability not a developmental disability?

The fact that a carte blanche editorial policy is in place at these journals also seems to be incentivising the submission of seemingly off-topic papers. If you use Web of Science to search within RIDD for the keywords “Alzheimer’s disease”, 22 papers come up. This is strange, because it’s difficult to see how you would define Alzheimer’s disease as a developmental disability and thus qualify for inclusion in the journal. The US National Library of Medicine (NLM) offers this definition, based on the American Psychiatric Glossary: Developmental disabilities are “disorders in which there is a delay in development based on that expected for a given age level or stage of development. These impairments or disabilities originate before age 18, may be expected to continue indefinitely, and constitute a substantial impairment.” Of Alzheimer’s disease, the NLM also has this to say: “AD usually begins after age 60”.

Here is where things get really interesting. Of those 22 Alzheimer’s papers in RIDD, 9 also refer to some other form of intellectual or developmental disability, so it’s easy to see why they might have been published in that journal. The other 13 don’t, and are all co-authored by some combination of Sigafoos, Lancioni and O’Reilly (let’s call these the SLO papers). We performed our own analysis (you can find the data here) , and for the non-SLO papers, the median time between the paper being submitted and being accepted was 57 days – in our experience, this seems about normal. For the 13 SLO papers, however, the median time between submission and acceptance was just 1 day. In three cases, papers were accepted on the same day that they were submitted.

Putting aside this extraordinary statistic, we asked Sigafoos and Lancioni whether they believed it was even acceptable to shoehorn papers relating to Alzheimer’s disease in what appears to be an inappropriate journal, and they responded that it fell within a “broader” definition of the term “developmental disability”:

While some of our RIDD papers did not focus on what some might see as the traditional types of childhood developmental disabilities, developmental disability could also be viewed from what we would see as a more contemporary life-span perspective. That perspective acknowledges that development occurs throughout the lifespan and thus a range of impairments or diseases can cause a disability that can affect development at any stage of life.

What’s really going on here? It’s difficult to see how these questionable practices reflect anything other than the gaming of an inappropriate editorial policy in order to publish more papers and boost personal citation metrics. The publishers of RIDD and RASD have commented only sparingly, but a statement from Michael Osuch, Publishing Director at Elsevier, indicates an awareness that something went wrong. These journal policies are now being changed, along with their editorial boards:

Our primary focus is on making the appointments of the editors on both RIDD and RASD, and to ensure rigorous peer review, with a minimum of two independent referee reports for all accepted articles. As you will already be aware, Dr Sebastian Gaigg from City University has agreed to take on the role of Editor-in-Chief on RASD and he is currently building an editorial team to support him on that journal (information to follow on the journal homepage shortly).

Elsevier should be credited with addressing the problem, but further questions need to be raised about the quality of the peer review process that papers already published in these journals went through, particularly those co-authored by the above researchers. If they were never fully reviewed, and only seen by Matson, then it is not unreasonable to suggest that these papers should be retracted. As we’ve said before, academic peer review isn’t perfect, but it is the least worst form of scientific governance we have.

Don’t shoot the messenger

Both Michelle Dawson and Dorothy Bishop should be applauded for bringing these issues to light. Being a whistle-blower is a difficult and thankless task, particularly when it involves questioning the integrity of other researchers. And of course, when questions are raised about dubious research practices, we must ensure that they are evidence-based and not founded in personal vendettas. In the present case, Bishop has been completely transparent. She has provided detailed data supporting her claims for anyone to scrutinise – you can download it freely from the Open Science Framework here.

Despite this, some commenters (mostly anonymous) have insisted on attacking Bishop and questioning her intentions, rather than addressing the real issue at hand. One commenter, “Sam”, has gone so far as to suggest that Bishop has deliberately misconstrued the data “with malicious intent”, referring to her as “Oxford Professor behaving badly” and encouraging others to complain both to the Vice Chancellor of Oxford University and the British Psychological Society.

This is shameful. It is one thing to have a legitimate concern about a situation in which one researcher is questioning the research practices of another, if that concern is well-founded. But resorting to ad hominem trolling is farcical, and should be called out as such. Thankfully, Bishop is resilient to these sorts of responses. “I’m not particularly surprised that they’re going after me”, she says. “People never just agree with you if you come out with critical comments; but if they don’t actually have much in the way of convincing arguments, then they just shoot the messenger”.

As a senior professor, Bishop is well protected from mudslinging but her experience highlights the difficulties that early career researchers face in similar situations. At such a vulnerable career stage, calling out potential malpractice by prominent members of your own research community could result in professional self-destruction, even if those criticisms are legitimate.

At the same time, the fact that we have to discuss the dangers of whistleblowing detracts from the central issue. Bishop has exposed what appears to be a major case of editorial malpractice at a number of academic journals, all seemingly happening under the publisher’s nose. For his part, Matson himself has remained vague on the matter. His comments to the Times Higher Education imply that he thinks that this is more an issue of “giving credit”. As the scrutiny of this case increases, this position is likely to become untenable.

Author disclosures: Chris Chambers is an associate editor at the Elsevier journal, Cortex.