By Erika Check Hayden

The 25,000 US women who give birth at home each year received shocking news from the nation's obstetricians early this year. Babies born at home die within their first month of life at two to three times the rate of children born in hospitals, the American Congress of Obstetricians and Gynecologists (ACOG) declared on the basis of a review published in July 2010.

But the study behind the warning is not as definitive as it seemed. Before the ACOG warning, the study generated so much criticism that the journal that published it, the American Journal of Obstetrics & Gynecology, was investigating it. The post-publication review documented errors in the original analysis, but it did not contradict the study or change the paper's conclusions, and the problems do not warrant a retraction, according to Elsevier in New York, the company that publishes the journal.

The ongoing debate over the study underscores the difficulty of conducting objective reviews on controversial medical topics such as home birth, says Andrew Vickers, a statistician at the Memorial Sloan-Kettering Cancer Center in New York.

"The scientific debate about home birth has become extremely polarized and politicized," Vickers says. "It is becoming hard to be anything but skeptical about anything but the most carefully conducted randomized trials."

The home-birth study was published last July by Joseph Wax, a specialist in maternal fetal medicine at the Maine Medical Center in Portland and his colleagues, who conducted a meta-analysis of 12 studies on home and hospital births. Using data from a subset of four of these studies, Wax's team concluded that babies born at home without birth defects were more likely to die in their first 28 days of life than those born in hospitals.

Soon after the study came out, epidemiologists, nurses, midwives, some obstetricians and home-birth advocates alleged that the paper did not meet standards set out in internationally recognized systems promoting best practice in conducting and reporting meta-analyses, such as the Cochrane Collaboration's Cochrane Reviews for evidence-based healthcare and the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) Statement.

Divergent methods

Researchers had also identified potential errors in the study's statistics. For instance, Carl Michal, a physicist at the University of British Columbia in Vancouver, Canada, identified an error in an online meta-analysis calculator used by Wax's team to arrive at some of its conclusions. The developer of that calculator acknowledged the error and has alerted potential users to the problem.

And Karin Michels, an epidemiologist at Harvard Medical School in Boston, Massachusetts, points out that Wax's team did not provide measurements of the variations between the studies included in its analysis; when studies diverge widely in methods and outcomes, they should not be combined, Michels says. As a result, she says, readers can't know how much the studies included may differ and whether it was appropriate to combine them.

In a statement sent to Nature, Elsevier says that in response to the criticisms, the journal enlisted three "specialists in maternal fetal medicine with expertise in meta-analysis and clinical research" to examine all the correspondence that the journal had received regarding the Wax paper. The reviewers attempted to reconstruct Wax's calculated risks for three outcomes: deaths of neonates, or infants from birth to 28 days old; premature births before 37 weeks of gestation; and 'postmature' births after 42 weeks of gestation, according to Elsevier. Postmaturity can carry risks such as diminished amniotic fluid. "In all 3 cases, the results the panel found were slightly different from that in the Wax paper," says Elsevier, although the panel did not find major differences in the risk estimates or the overall statistical significance of the results.

The review committee recommended that the journal publish full summary graphs online for each outcome in the study, "which will allow readers to better assess the study findings", says Elsevier. But the committee did not recommend a retraction of Wax's paper by the journal. "There were a number of issues raised in the letters, many of which the panel felt were subjective and should be debated openly," Elsevier told Nature.

On 28 February, Wax's team posted the requested summary graphs on the journal website. The risk of newborn death and postmaturity among babies born at home is now higher than it was in the original paper, and the risk of prematurity is now lower. The document does not discuss whether or how Wax's group erred in its original calculations, or what changes were made to produce the new results.

Controversial exclusion

Critics are not appeased, because many had argued that Wax's team erred by inappropriately including or excluding studies from some of these outcomes in the first place. Epidemiologists consulted by Nature, who are not involved in the home-birth debate, agreed that there were problems with the study design.

Diana Petitti, an epidemiologist at the Arizona State University Center for Health Information and Research in Phoenix who authored a book on meta-analysis, says Wax's group should not have excluded data from a major Dutch study, published in 2009, that examined more than 300,000 home births for many outcomes, including the risk of newborn deaths. That study found no increased risk of death after home birth in the first week of life.

Petitti says this issue could have led to incorrect conclusions even if the statistical methods were sound. "The problem of excluding the Netherlands study dwarfs any problem related to software or the statistical models," she says.

ACOG and Wax declined multiple requests from Nature for comment. But the journal will publish more letters regarding the study and responses from Wax in coming weeks.

Elsevier says the journal did not notify ACOG about the investigation of Wax's study. Nonetheless, the substantial public criticisms of the paper should have given ACOG "some reservations" about using the study in its statement about home births, said David Moher, a clinical epidemiologist at the University of Ottawa and chair of the PRISMA Statement.

But, Moher says, it is unfortunately common for reviews to contain major methodological flaws.

"The quality of non-Cochrane Reviews leaves much to be desired," he says.