We’re going to have to do something about peer review. Posted by Pointman on February 15, 2013 · 13 Comments

Not too many years ago, there were two researchers who discovered that stomach ulcers were not the result of a stressful lifestyle, but simply an infection. The idea that ulcers were related to a high stress lifestyle was so deeply embedded in the medical establishment psyche, they couldn’t get a fair hearing, never mind getting their work published. To make that happen, one of them did an extraordinary courageous thing; he infected himself with the germs he knew would give him a stomach ulcer, got the subsequent condition unambiguously diagnosed, took the requisite medication and was cured.

Now, that’s what I call having confidence in your research.

Rather than painful decades of eating a bland diet, changing your lifestyle or getting pieces of your stomach progressively cut out, a simple course of drugs will now cure a stomach ulcer, which is why you don’t hear much about them nowadays. Thirty years down the line, even the most advanced medicine tends to look like nothing better than butchery.

That story is illustrative of some fundamental things about science. For starters, it’s never settled. It’s for that reason, if you can’t honestly examine a doubt about any theory, you’re doing belief rather than science. Most theories are superseded by a better one or are found out to be just special cases of a higher level abstraction. The Newtonian view of celestial mechanics giving way to Einstein’s work, would be the classic example.

Of course, some theories are just plain wrong to the point of being junk science. Well-known examples of the latter would be Phrenology, Lysenkoism and Eugenics, which were all essentially outbreaks of pathological science. The sobering thought is that all of them were thought of as mainstream science for decades.

Some theories are the result of bad science; the prime example of this being the Ptolemaic system, which placed the Earth at the centre of our solar system, and while it was totally wrong, it still gave some functionally accurate results, which is why it survived as mainstream thought for fifteen hundred years.

How can something that badly wrong, be believed in for a millennium and a half, by the educated people who would be the equivalent of today’s scientists?

There are two reasons. The first one is that if you’re not familiar with a particular field, the natural tendency is to assume that those who are experts in it, know their stuff. You accept their assessment of any new theories in their area. The second reason is that people who know the area well, are nearly always heavily invested in the current status quo. They’ve spent years studying a particular field and become unquestioning of its central tenets. Expecting them to concede something they’ve backed for most of their adult life was wrong, is a big ask. Anyway, by that stage, their intellectual arteries have hardened. It’s natural for them to be wary of and resist any seismic shifts in the fundamental theories underpinning it.

Given those barriers to change, how does science actually manage to keep evolving? The answer is publication of papers by researchers. The problem with this, if you’re coming up with research which is a real game changer, is the peer review system. I wrote a previous article which outlined the peer review process and touched on some of the problems associated with it, which you can find here, but if you’re unfamiliar with the process, the following paragraph from the piece summarises it.

A scientist does some research, writes up the results in the form of a paper and submits it to a scientific journal for publication. Whether it’s published or not, is an editorial decision. The editor, who obviously can’t be an expert in all branches of science, has to seek advice from other experts in the field the paper is dealing with. For each area, the editor will have two or three experts he can ask to review or referee the paper. Some journals ask the author for a list of potential reviewers. The reviewers are usually unpaid and working scientists themselves.

The peer reviewers will come back with one of four recommendations; publish it, publish after comments have been addressed, reject it outright or reject it for resubmission after rework at a later date. Obviously if the paper undermines the life’s work of the reviewers, it’ll be lucky to get the go ahead from them. That process is called gatekeeping or excluding theories that contradict ones you support. The climategate emails illustrated not only gatekeeping, but a complete subversion of the whole peer review process in climate science. The reviewers were determined that papers they disliked were not going to be published, even if that meant “redefining the whole peer review process.”

Once pre-publication peer review became common in the 1950s, it immediately pitted innovation in a particular science against the establishment of that area, and it’s a David and Goliath situation. Before considering a possible solution allowing more innovation and freedom of thought, it’s worth examining how innovation came about in the centuries before the straight jacket of the peer review process was put in place, because I think it gives a good indication of the way forward.

Quite simply, people wrote books or papers. There was nobody gatekeeping; controlling what you could and could not put out into the world. Some ideas were so bad, they sank without comment. Other people wrote their own books and papers, discussing your ideas. Beneath all that public discussion, spirited conversations between the interested parties occurred, usually by letter. As a system that encouraged active debate of new innovative theories, it worked better than peer review. Once you get past the paper dependency of the whole process, there’s a certain latter-day familiarity about the whole thing.

In areas of science subject to controversy, we’ve already begun to move over to the equivalent twenty-first century implementation of the old system using the blogosphere. Papers are published online and people discuss them, either on their own blogs or directly as comments underneath the paper. Just as in the old system, there’s a Darwinian selection process going on; loony papers sink without trace, interesting ideas get examined by a lot more minds and from a basis of different skill sets.

In fields like climate science, where airing controversial views can threaten your career or if you’re a student, who knows nonconformity will threaten your degree, it offers the protection of anonymity, both to authors and the commenters.

This isn’t my suggestion of a way forward; it’s simply my recognition of the direction we’re already heading towards in the controversial areas of science. What’s driving it is quite simple. When people are prevented from publishing in the mainstream science journals, they’ll publish on the internet. When sensible commentary on dubious science is heavily censored, that commentary moves out to the freedom of the internet. Once moved onto the internet, it won’t be coming back anytime soon, which is one of the reasons why circulations of what used to be the popular science journals, are plunging.

This change in how new ideas in science are debuted and debated is happening and it is the future, whether you’re comfortable with it or not. Novy mir.

©Pointman

Related articles by Pointman:

The decline of popular science journals.

Is climate science just a belief?

Global warming and pathological science.

How climategate destroyed the science of global warming.

Click for a list of other articles.