NASA's arsenic debacle tells us a lot about what's wrong about the relationship between science, peer review and the media in the 21st century.

I've been following NASA's arsenic-munching microbe story for several days now, biding my time while the storm played out. In a way, the science itself is the least interesting part of the affair. What's much more interesting is that the drama has given us an opportunity to see how a collection of related problems in different areas of science outreach can combine to seriously damage the credibility of a highly-respected scientific institution, and by extension science itself.

The affair that we have somehow managed to avoid calling 'Arsenicgate' is, in essence, a story of everything that's wrong about the relationship between science, peer review, the world of publishing, and the mainstream and independent branches of the media in 2010. In the rest of this post I want to try to pick apart the roles of the various people involved.

The Researchers (Wolfe-Simon et al)

I'm not particularly qualified to comment on the specific science involved, but the emerging consensus among many scientists in the field is that at best the research isn't good enough to support the weight of its own conclusions. Some have suggested that the work is so profoundly flawed that it shouldn't have even been published in the first place.

But to focus on the researchers would be to miss the point really. There may be errors or methodological weakness in their work, but it's not a crime if Wolfe-Simon et al write a paper that sucks. The whole point of submitting a paper for publication is to solicit feedback, and it's not their fault that a) the paper was accepted or b) the paper was sucked into the hungry input nozzles of NASA's PR machine.

The Journal

Having been written, the paper was published in the prestigious journal Science. The best comment about them comes from my Guardian colleague Alok Jha:

"Science has finally launched the paper that started all the fuss. Unfortunately only the abstract has been made available to non-subscribers (it's $15 to access the article for 24 hours)."

The term 'publish' when used in science is nearly always accidentally-ironic, since most work isn't made public at all, but kept behind the ubiquitous pay-walls that plague the free exchange of scientific ideas.

We can debate that general problem another time, but it is absolutely beyond ridiculous that a paper hyped as being of massive public interest was not made freely available to the public. Or indeed to anyone at the time the embargo ended - we had to wait for hours after the news broke before the paper was even published at all, making it impossible to evaluate the accuracy of the reports. I think it's disgraceful, and the people involved should be ashamed of themselves.

Update (18:54): Ginger Pinholster from AAAS has e-mail me to say that the paper has now been made freely available to the public, which is excellent news and something I think they should be congratulated for doing, and encouraged to keep doing in the future. Ginger also noted that staff were caught on the hop when the embargo was ended earlier than planned.

Should the paper have been published in the first place? Carl Zimmer's blog post for Slate collects the responses of numerous scientists to the work, including the University of Colorado's Shelley Copley declaring that: "This paper should not have been published."

There are two distinct questions here to tease apart: 'should the paper have been published?' and 'should it have been published in Science?'

To the first question I would say 'yes'. Peer review isn't supposed to be about declaring whether a paper is definitely right and therefore fit for publication on that basis. The purpose of publishing paper is to submit ideas for further discussion and debate, with peer review serving as a fairly loose filter to weed out some of the utter crap. The contribution a paper makes to science goes far beyond such trivialities as whether or not it's actually right.

Wolfe-Simon et al's paper might be wrong, but it has also sparked an interesting and useful debate on the evidence and methodology required to make claims about this sort of thing, and the next paper on this subject that comes along with hopefully be a lot stronger as a result of this public criticism. You could argue on that basis that its publication is useful.

I would argue that the real bone of contention is whether it should have been published in Science - after all, if it had appeared in the Journal of Speculative Biological Hypotheses (and not been hyped) nobody would have given a crap. On this I'm not really qualified to comment, but what I can say is that given the wealth of scientists coming forward to criticize the work, it's remarkable that the journal found three willing to pass it.

In any case, Science accepted the paper, and so the hype began.

The Announcement

In the week before the paper was published, NASA, Science and the media became locked in a bizarrely awkward dance. The first hint of the story from NASA came from their initial statement/trail:

"NASA will hold a news conference at 2 p.m. EST on Thursday, Dec. 2, to discuss an astrobiology finding that will impact the search for evidence of extraterrestrial life."

The story itself was now under an embargo, so this achieved two things: it ensured that people were going to start speculating wildly about the announcement for several days, and that the speculation would fixate on alien life. How either of those things helped the public understanding of science, I'm not really sure.

NASA sent a copy of that statement to Science for approval before releasing it, and Ivan Oransky has highlighted an interesting comment from a spokeswoman for AAAS, the publishers of Science, accepting that in hindsight they should have flagged the phrase 'extraterrestrial life'.

A much bigger problem though was that the news, supposedly now under embargo, had already leaked into the public domain. This became clear when Paul Sutherland, a freelance reporter working for The Sun, apparently 'broke' the embargo the day before it was due to be lifted. In fact, as Oransky reports, he had simply pieced together the story from information already in the public domain:

"Despite wild and rampant speculation on the web, it was clear that the conference was unlikely to be about a discovery of extra-terrestrial life, for the reasons very well put forward by Stuart Atkinson here. But as with Mars, I simply checked out the research interests and activities of the scientists who would be on NASA's conference panel. In particular, I found that Felisa Wolfe-Simon had been looking for life that was dependent on arsenic in Mono Lake, California. Earlier this year Felisa said in an interview that she had exciting results that she was not yet able to reveal. I figured that NASA were hardly going to call a top-flight conference for her to say she had found nothing!"

At this point, the entire exercise degenerated into a farce. Sutherland noted he was "glad that, free from the shackles of the embargo system, we were able to balance some of the speculative nonsense out there," but his comment is undermined by his hilarious statement (NB: actually this wasn't Paul's statement, but a sub-editor's - see his comment here) in the article that:

"It opens up the possibility that Extra Terrestial aliens like the one from the 1982 movie CAN exist in the solar system."

Meanwhile AAAS whined impotently about "erroneous tabloid speculation." Eventually the 'official' announcement arrived, and later still the paper, but by then the story had already lost a lot of its shine. A day-old story about arsenic-eating microbes isn't nearly as sexy as the alien from a Spielberg moving turning up on Titan.

The Blogosphere

The blogosphere is where this farcical handling of science outreach collided viciously with reality. What's remarkable is that it was only at this stage that the checks and balances that ought to have prevented things going this far finally came into play, and when they did it was like watching a blob of cream cheese hit a metal grater at five million miles per hour.

The quality, accuracy and context of material available on leading blogs exceeded that of much of mainstream media reporting by light years. While newspapers ran away with the story, it was left to bloggers like Ed Yong, Carl Zimmer, Lewis Dartnell and Phil Plait to put things into perspective.

But more importantly it turns out that peer review is being done on blogs. John Hawks and Alex Bradley - both scientists with relevant expertise - found methodological problems. Rosie Redfield, a microbiology professor a the University of British Colombia, wrote an extensive and detailed take-down of the paper on her blog that morphed into a letter to Science, which I sincerely hope they publish.

Carl Zimmer then went one step further, and organized his own peer review process, recruiting a dozen scientists to give their opinions on the paper.

Not only is this clearly peer review, it's also peer review that managed to beat the efforts of one of the most prestigious journals in science. That has some serious implications for the future. It would be daft to talk about blogs taking over peer review, but clearly having lots of scientists discussing papers through social networks can achieve more than the current system, at least some of the time.

The Response to the Response

At this point, NASA and AAAS had a fantastic opportunity to salvage something from the ruins. An effective PR response could have recognized that mistakes were made and welcomed and engaged with the vigorous debate occurring online.

Apropos of nothing, here's Dwayne Brown, senior public affairs officer in the office of communications of the NASA Science Mission Directorate, talking to Ivan Oransky:

"The real issue is that the reporting world has changed because of the Internet/bloggers/social media, etc. A "buzz" term like ET will have anyone with a computer put out anything they want or feel. NASA DID NOT HYPE anything – others did. Credible media organizations have not questioned NASA about any text. Bloggers and social media have……..it's what makes our country great—FREEDOM OF SPEECH"

Oh dear. It's never a good sign when your senior PR person speaks in 'MORON CAPITALS'. The outpouring from Dwayne continued in a statement to CBC news:

"When NASA spokesman Dwayne Brown was asked about public criticisms of the paper in the blogosphere, he noted that the article was peer-reviewed and published in one of the most prestigious scientific journals. He added that Wolfe-Simon will not be responding to individual criticisms, as the agency doesn't feel it is appropriate to debate the science using the media and bloggers. Instead, it believes that should be done in scientific publications."

It is a spectacularly arrogant statement to make, pretty much the equivalent of sticking your fingers in your ears and screaming 'LA LA LA LA LAAAA'. David Dobbs provides probably the best response to:

"This is a call to pre-Enlightenment thinking. Brown is telling us to judge utterances not by their content, not even by the integrity, reputation, and experience of the individuals who deliver them, but by whether they're delivered from the proper place in the proper building — in pre-Enlightenment days, the Church of Rome; in Brown's post-arsenic days, the Church of the Peer-reviewed Journal. "It's an extraordinary dismissal. Rosie Redfield is a full-bore member of the academy and a researcher in the field under question. She is — to extend the metaphor — a priest. But though Redfield wears the proper robes, Brown wants to dismiss her because she's not standing on the proper altar."

Aside from that, the attitude is hypocritical, as Jonathan Eisen neatly pointed out to Carl Zimmer:

"If they say they will not address the responses except in journals, that is absurd," he said. "They carried out science by press release and press conference. Whether they were right or not in their claims, they are now hypocritical if they say that the only response should be in the scientific literature."

Dwayne may not like blogs, but they have just schooled the collective might of NASA and AAAS on how to do science. And if you live by the media, you have to be prepared to die by the media.

Conclusions

Screwing up is understandable, but when you repeatedly screw up the same way you've got a problem.

At almost every stage of this story the actors involved were collapsing under the weight of their own slavish obedience to a fundamentally broken... well... 'system' is the right word, but I find myself toying with 'ideology'.

The journal system prevented the public from accessing the paper. Peer review failed. The research was over-hyped in NASA's original 'sphinx-like' press release. An embargo was enforced on information that had already leaked into the public domain, and even as speculation mounted news outlets were barred from reporting the facts. The paper itself wasn't even available until hours after the embargo lifted, and when the research was finally published, and scientists began to criticize it, NASA's press people issued a spectacularly ill-mannered and arrogant response.

What makes this fiasco so infuriating isn't that people screwed up, it's that they screwed up in the same entirely obvious and predictable ways that they keep screwing up, yet still apparently refuse to change or reform any part of the process, as if they are somehow resigned to the tragedy of inevitable, banal failure.

It's not the first time this has happened. It won't be the last time either. I think it's all a bit silly really.

layscience@googlemail.com | @mjrobbins