Trial By Error: Our Exchange of Views with BMJ Open

By David Tuller, DrPH

Last week, Professor Racaniello e-mailed a letter of concern signed by more than a dozen experts to Dr. Trish Groves, editor-in-chief of BMJ Open. The letter involved Professor Esther Crawley’s school absence study, which the journal published in 2011. As I’d documented in a post last year, the study exempted itself from ethical review based on the false claim that it was “service evaluation” and not “research.”

The next day, Dr. Groves sent the following e-mail:

Dear Professor Racaniello and colleagues

Thank you for your letter.

We assume that you know this case has been thoroughly reviewed by BMJ Open and has been considered by the Committee on Publication Ethics.

If you would like us to also consider posting an eletter to the paper at BMJ Open, please submit it by clicking the link next to the article.

With best wishes

Trish Groves

**********

Needless to say, this was not a satisfactory answer. Professor Racaniello e-mailed the following response today:

Dear Dr. Groves:

Thank you for your response. We are aware that BMJ Open reviewed the issue last year after concerns were raised about the paper. That review might have been “thorough” from your perspective, but it was nonetheless deficient.

An objective reading of the school absence paper would reveal that it does not meet accepted definitions of “service evaluation,” as our letter noted. The data for this assessment of a “pilot clinical service” were collected in face-to-face meetings arranged specifically for the purposes of the study, which featured a hypothesis and generalizable conclusions. Given these facts, the only reasonable interpretation is that the study qualified as “research” and required ethical review, according to the guidelines promulgated by the Health Research Authority.

We are also aware that BMJ Open submitted this case to the COPE Forum, since Dr. Tuller analyzed your submission and the Forum’s response for Virology Blog. As he reported and as we noted in our letter, the Bristol University statement that BMJ Open has cited in its defense involved an anonymous national CFS/ME database and had nothing to do with the first-hand data collected for the school absence study.

BMJ Open’s submission to the Forum also quoted from a letter from the research ethics committee purportedly backing the investigators’ claim that the study did not require ethical review. In fact, the quote was apparently from the original 2007 research ethics committee letter cited in the paper. Like the Bristol University statement about the national CFS/ME database, this 2007 letter had nothing to do with the school absence study. BMJ Open’s thorough review of the matter must have missed these critical points.

In its response to the BMJ Open submission, the COPE Forum noted that the journal did not provide full details about the study methodology. Here’s the main part of the Forum’s response:

“The Forum suggested that perhaps the issue is not whether or not the service evaluation is research, but was the evaluation carried out in human subjects (which would require a sound ethics approach) or were the data contained in registries where the patient data were anonymised. It would appear that the latter is the case and that this is a secondary data analysis, but the editor could ask for clarification from the author on the methodology as it needs to be adequately described. Was this a dataset developed out of a research project that had ethics approval for human subjects? If so, the secondary analysis might not need new ethics approval if additional analyses were covered in the initial approval. The methodology is confusing the issue of whether ethics approval was required. The Forum suggested these points need to be clarified before a decision on whether to add a correction on the article or to respond to the blogger.”

The Forum’s message is not complicated: If the study was “carried out in human subjects,” it would “require a sound ethics approach.” The response further indicates that the Forum has tentatively assumed—wrongly, as it turns out–that the study did not involve “human subjects” but only “registries where the patient data were anonymized.” To resolve the uncertainty, the Forum suggests that the journal editor “ask for clarification from the author on the methodology.” The Forum reiterates this point, noting that “the methodology is confusing the issue of whether ethics approval was required,” and once again suggests that the journal needs to clarify these points before a decision on a correction can be made.

As far as we know, BMJ Open has not yet complied with the Forum’s suggestion that the journal seek “clarification” of the school absence paper’s methodology. As our letter noted, obtaining such clarification would only require that you and other BMJ Open editors read the actual paper. The investigators themselves provided definitive evidence in their description of the methodology that this was not “service evaluation” but “research.”

In short, BMJ Open published a study involving children that wrongly exempted itself from ethical review. The case is fully documented, and it is indisputable. It is incumbent upon BMJ Open to take appropriate action and address the problem.

Sincerely—

Vincent R. Racaniello, PhD

Professor of Microbiology and Immunology

Columbia University

New York, NY, USA

Ronald W. Davis, PhD

Professor of Biochemistry and Genetics

Stanford University

Stanford, CA, USA

Jonathan C.W. Edwards, MD

Emeritus Professor of Medicine

University College London

London, UK

Rebecca Goldin, PhD

Professor of Mathematics

George Mason University

Fairfax, VA, USA

Leonard A. Jason, PhD

Professor of Psychology

DePaul University

Chicago, IL, USA

Steven Lubet

Williams Memorial Professor of Law

Northwestern University Pritzker School of Law

Chicago, IL, USA

Marlon Maus, MD, DrPH, FACS

Director, DrPH Program

School of Public Health

University of California, Berkeley

Berkeley, CA, USA

Patrick E. McKnight, PhD

Professor of Psychology

George Mason University

Fairfax, VA, USA

Philip B. Stark, PhD

Professor of Statistics

University of California, Berkeley

Berkeley, CA, USA

John Swartzberg, MD

Clinical Professor Emeritus

School of Public Health

University of California, Berkeley

Berkeley, CA, USA

Samuel Tucker, MD

Assistant Clinical Professor of Psychiatry (retired)

University of California, San Francisco

San Francisco, California, USA

David Tuller, DrPH

Senior Fellow in Public Health and Journalism

Center for Global Public Health

University of California, Berkeley

Berkeley, CA, USA

William Weir, FRCP

Infectious Disease Consultant

London, UK