Trial By Error: A Letter to Archives of Disease in Childhood

By David Tuller, DrPH

Earlier today, Professor Racaniello e-mailed the following letter to Nick Brown, the editor-in-chief of Archives of Disease in Childhood, one of the journals from the BMJ Group. Archives recently published a study of the Lightning Process in kids with ME/CFS.

**********

Dear Dr. Brown:

In September, Archives of Disease in Childhood published a study called “Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial” [1]. Like many studies of this illness, it was an open-label trial with self-reported outcomes—a study design known to be highly vulnerable to bias.

The researchers have received criticism for their initial decision to conduct a pediatric trial of the Lightning Process–a commercial, pseudo-scientific intervention that combines elements of life-coaching, osteopathy, and neurolinguistics programming. The intervention encourages participants to report that it has made them better, another source of potential bias. Beyond the decision to study the Lightning Process, the study suffered from major cross-contamination between treatment arms and significant loss-to-follow-up, among other problems.

In addition to these issues, we are also concerned about a methodological anomaly reported on Virology Blog by David Tuller from the University of California, Berkeley: An earlier feasibility trial was extended into the full trial even as the researchers swapped primary and secondary outcomes following a review of the feasibility trial results [2]. The researchers received ethical approval for these study design changes but failed to mention them in the Archives paper, although the changes enabled the findings to be presented in a much more positive light than would otherwise have been the case.

These changes are only apparent from a review of other trial-related documents, such as the registration [3], the feasibility trial protocol [4] and the full-trial protocol [5]. Our specific concerns include the following:

*The researchers registered the study as a prospective trial. But more than half the sample were apparently participants recruited for the feasibility trial [6], starting almost two years before the registration date. The Archives paper did not mention this fact or explain the discrepancy in dates.

*In the feasibility trial protocol, school attendance at six months was the primary outcome, and self-reported physical function at six months was a secondary outcome. After reviewing results from the feasibility trial, the researchers swapped these outcome measures for the full trial—school attendance at six months became a secondary outcome, and self-reported physical function at six months became the primary outcome. The Archives paper did not disclose the outcome-swapping that occurred after more than half the sample had provided data as participants in the feasibility trial.

*As its main finding, the Archives paper highlighted the reported benefits in the Lightning Process group for self-reported physical function at six months, the new primary outcome. Self-reported school attendance at six months, now a secondary outcome, yielded null results and received scant mention in the Archives paper. Thus the outcome-swapping allowed the researchers to report positive results for their primary outcome. Had they not switched the outcomes after more than half the sample had provided data, they would have had to report null primary outcome results

*In both the feasibility trial protocol and the full-trial protocol, the researchers promised to seek official school attendance records as a complement to self-reported school attendance. The Archives paper did not provide official school attendance records, even though it highlighted benefits at 12 months in self-reported school attendance. The researchers offered no explanation for the absence of these official school attendance records. Interestingly, it was stated in a recent news account that the official school attendance records corroborated the self-reported school attendance results, suggesting that these data were in fact collected.

*Not surprisingly, the media coverage of the Archives paper focused on the reported benefits in the Lightning Process arm for the revised primary outcome of self-reported physical function at six months. The null results for school attendance at six months, the original primary outcome, were largely ignored. The outcome-swapping that occurred after more than half the sample had provided data—and that the Archives paper did not mention—ensured that the reported findings received more positive media coverage than they otherwise would have.

Transparency is essential in scientific research. In this case, the Archives paper failed to disclose salient and relevant information about the composition of the study sample, the outcome-swapping, and the official school attendance records. These missing details, if provided, would help others to more properly assess the validity and reliability of the reported findings.

The Archives paper must be corrected to acknowledge that the sample included the feasibility trial participants and that the review of their data informed the decision to switch the outcome measures. The paper should also present separate results for participants in the feasibility trial and those recruited afterwards. And the corrected version must clarify the status of the official school attendance records; if these data were collected, as the recent news account suggested, they should also be reported.

We urge Archives of Disease in Childhood to take these concerns seriously and address them promptly.

Sincerely—

Vincent R. Racaniello, PhD

Professor of Microbiology and Immunology

Columbia University

New York, NY, USA

Lisa F. Barcellos, MPH, PhD

Professor of Epidemiology

School of Public Health

California Institute for Quantitative Biosciences

University of California, Berkeley

Berkeley, CA, USA

Robin Callender Smith, PhD

Professor of Media Law

Centre for Commercial Law Studies

Queen Mary University of London

Barrister and Information Rights Judge

London, UK

Lily Chu, MD, MSHS

Independent Consultant

Burlingame, CA, USA

Ronald W. Davis, PhD

Professor of Biochemistry and Genetics

Stanford University

Stanford, CA, USA

Jonathan C.W. Edwards, MD

Emeritus Professor of Medicine

University College London

London, UK

Valerie Eliot Smith

Barrister and Visiting Scholar

Centre for Commercial Law Studies

Queen Mary University of London

London, UK

Rebecca Goldin, PhD

Professor of Mathematics

George Mason University

Fairfax, VA, USA

Leonard A. Jason, PhD

Professor of Psychology

DePaul University

Chicago, IL, USA

Michael W. Kahn, MD

Assistant Professor of Psychiatry

Harvard Medical School

Boston, MA, USA

Nancy Klimas, MD

Director, Institute for Neuro-Immune Medicine

Nova Southeastern University

Director, Miami VA Medical Center GWI and CFS/ME Program

Miami, FL, USA

Bruce Levin, PhD

Professor of Biostatistics

Columbia University

New York, NY, USA

Steven Lubet

Williams Memorial Professor of Law

Northwestern University Pritzker School of Law

Chicago, IL, USA

Marlon Maus, MD, DrPH, FACS

Director, DrPH Program

School of Public Health

University of California, Berkeley

Berkeley, CA, USA

Patrick E. McKnight, PhD

Professor of Psychology

George Mason University

Fairfax, VA, USA

Zaher Nahle, PhD, MPA

Vice President for Research and Scientific Programs

Solve ME/CFS Initiative

Los Angeles, CA, USA

Philip B. Stark, PhD

Professor of Statistics

University of California, Berkeley

Berkeley, CA, USA

John Swartzberg, MD

Clinical Professor Emeritus

School of Public Health

University of California, Berkeley

Berkeley, CA, USA

David Tuller, DrPH

Senior Fellow in Public Health and Journalism

Center for Global Public Health

University of California, Berkeley

Berkeley, CA, USA

William Weir, FRCP

Infectious Disease Consultant

London, UK

Carolyn Wilshire, PhD

Senior Lecturer

School of Psychology

Victoria University of Wellington

Wellington, New Zealand

**********

[1] Crawley EM, Gaunt DM, Garfield K, et al. 2018. Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial. Archives of Disease in Childhood; 103(2):155-164

[2] Tuller D. 2017. Trial by error: the SMILE trial’s undisclosed outcome-swapping. Virology Blog; Dec 13. Accessed at: https://www.virology.ws/2017/12/13/trial-by-error-the-smile-trials-undisclosed-outcomes/

[3] ISRCTN Registry: ISRCTN8145620. July 31, 2012. Specialist Medical Intervention & Lightning Evaluation: Comparing specialist medical care with specialist medical care plus the Lightning Process for Chronic Fatigue Syndrome or Myalgic Encephalopathy (CFS/ME). Accessed at: http://www.isrctn.com/ISRCTN81456207

[4] Crawley EM, et al. July 2010. SMILE Protocol v6: Assessing the feasibility and acceptability of comparing the Lightning Process with specialist medical care for Chronic Fatigue Syndrome or Myalgic Encephalopathy (CFS/ME)–pilot Randomised Controlled Trial. Accessed at: http://www.bristol.ac.uk/media-library/sites/ccah/migrated/documents/smprotv6final.pdf

[5] Crawley EM, Mills N, Hollingworth W, et al. 2013. Comparing specialist medical care with specialist medical care plus the Lightning Process® for chronic fatigue syndrome or myalgic encephalomyelitis (CFS/ME): study protocol for a randomised controlled trial (SMILE Trial). Trials; 14:444

[6] Crawley EM, Mills N, Beast L, et al. 2013 The feasibility and acceptability of conducting a trial of specialist medical care and the Lightning Process in children with chronic fatigue syndrome:: feasibility randomized controlled trial (SMILE study). Trials; 14:415.