There is no good evidence that Yersinia pestis—the bacterium that causes plague—is riding aboard the New York City subway. That’s the message from several microbiologists, in response to a wave of news stories that emerged last Friday.

“Plague, anthrax and cheese? Scientists map bacteria on New York subway,” said the Guardian. “From beetles to bubonic plague: Bizarre DNA found in NYC subway stations,” proclaimed the Washington Post. “Terrifying microbe map of New York’s subway system reveals superbugs, anthrax and bubonic plague,” blathered the Daily Mail, duly retaining its crown as the champions of scaremongering.

All of these stories were based on a census of some of New York City’s smallest residents—its microbes. Over the course of 18 months, a team of scientists led by Christopher Mason from Weill Cornell Medical College swabbed surfaces all over the city, including every open subway station (see a video about their subway sampling here). They then analysed the DNA in their samples to identify the microbes that lived on each surface.

There were plenty of interesting results. For example, the bacteria from South Ferry Station, which was flooded by Hurricane Sandy in 2012, were closer to marine microbes than to those in the rest of the subway. But the team’s most notable claims were that they found DNA from the plague bacterium Yersinia pestis in three samples, and from the anthrax bacterium Bacillus anthracis in two. These nuggets predictably wound their way into every major news story.

The researchers downplayed the significance of these results, saying that if Y.pestis was present, it was unlikely to be “active and causing disease in people”. As Mason told the New York Times, “We’re saying there’s evidence for these things… but no one should worry.”

But several microbiologists think that even this statement goes too far. “[They] have not provided persuasive evidence that the agents that cause plague or anthrax are present anywhere in the New York City subway system,” says Ian Lipkin, a well-respected virus hunter from Columbia University. “The genetic footprints they report are not specific for the agents that cause anthrax or plague; they are also found in other common bacteria that are not associated with disease.”

The team arrived at their conclusions after compiling the DNA in each sample, breaking it down into smaller pieces, and then sequencing the fragments. They then searched for these sequences—or “reads”—in a public database of genes from all known organisms. If they found many matches for a given microbe, they concluded that said microbe was present in their samples.

“This method is notoriously unreliable,” says Willem van Schaik from Utrecht University, who studies antibiotic-resistant bacteria. As Lipkin also noted, it’s prone to false alarms, because a given read could match DNA that’s found in many other bacteria besides Y.pestis.

Rob Knight, formerly at the University of Colorado Boulder, showed just how ludicrous this problem can get last year. His colleague, Andrea Ottesen at the FDA, swabbed tomato plants in a field in Virginia, and Knight analysed the DNA in those samples. He found matches to the duck-billed platypus—an Australian animal, not known to live in Virginia. They then analysed over 19,000 publicly available microbiome samples from around the world; around a third threw up matches for platypus DNA. Either the platypus secretly rules the world or, more likely, this was a hilarious case of false positives gone mad. The team, including Antonio González Peña, even created a programme called Platypus Conquistador to rectify the problem.

There are signs of similar problems in the subway paper. In the case of Y.pestis, the team found several reads that matched a plasmid—a free-floating ring of DNA that sits outside the bacterium’s main genome. They highlighted one particular section of the plasmid in one of their figures. Based on this, Van Schaik did his own search and found that sequences in this section are also found in at least three other bacterial species.

Mason’s team also identified DNA from many eukaryotes—that’s animals, plants, and other complex organisms—and listed the top species in one of their tables. They were, starting from the top: the mountain pine beetle, which lives in the west coast of North America; the Mediterranean fruit fly, which does not exist in the continental US; the cucumber; and humans. Hmm. The fly, in particular, is a major agricultural pest and the subject of intensive surveillance. Its presence in New York is “so unlikely that I think the approach they used is just flawed,” says Van Schaik.

Unlike the fly, Y.pestis does exist in the US, but in the southwest where it infects rodents. It is a stranger to New York. Lipkin’s team have done extensive surveys of the city’s rats and failed to find Y.pestis in any of them.

To convince their critics, Mason’s team would have to show that reads from the subway analysis map to the entirety of the Y.pestis plasmid. Then again, plasmids can easily move from one bacterium to another, so it would be even more convincing to show reads that map to Y.pestis’s actual chromosome. “Even better would be to prove the existence of Y. pestis through some independent means, such as culture,” says Nick Loman from the University of Birmingham, who studies the genomes of disease-causing microbes. By “culture”, he means trying to actually grow bacteria from collected samples.

I contacted Chris Mason about these criticisms and he is preparing a blog post to address them, and others that he has received. [Update 18 Feb: Mason has published the post and it is humble, informative, detailed, and introspective]

As I mentioned, the paper has other interesting results and it might seem churlish to pick on this one. But it symbolises some of the problems in the study of microbiomes—the collections of bacteria that live in specific animals or environments. Microbiome research is among the hottest fields in biology and is attracting hordes of enthusiastic scientists. This is great—(disclosure: I’m writing a book on animal microbiomes)—but the field also risks throwing up a lot of misleading and false conclusions if methods aren’t applied properly and results aren’t analysed cautiously. (Last November, I wrote about another common problem that might lead to false positives in a lot of microbiome studies.)

One could argue that these issues come out in the wash, and that science corrects itself. Indeed, Mason told the New York Times that not reporting the fragments of anthrax and plague “would have been irresponsible”. Then again, readers were hit with a wave of headlines that raised the possibility of plague on the subway. “If I wasn’t a microbiologist, I would be scared by this and rightfully so,” says Van Schaik.

Marc Lipsitch, an epidemiologist from the Harvard School of Public Health, agrees, and adds that panic-quelling follow-up stories, like this one, don’t help matters. “Stories that present a highly hedged finding as news, but then say “Don’t worry”, make scientists seem like we aren’t telling the whole truth,” he says. “Either it is news, or we shouldn’t worry, but it’s tough to see how both could be true.”

What Are You Touching in NYC’s Subway?

Update: Nick Loman has put up a short post illustrating one of the problems I talked about in this post. He took E.coli, shred its DNA, sequenced the pieces, and then matched them up to databases. By right, 100% of the reads should come out as E.coli. In fact, just 61% of them did.

Reference: Afshinnekoo et al., Geospatial Resolution of Human and Bacterial Diversity with City-Scale Metagenomics. 2015. Cell Systems http://dx.doi.org/10.1016/j.cels.2015.01