Bertrand Russell's essay "On the Value of Scepticism" was published 85 years ago this year. Its opening lines are as relevant today as they were then: "I wish to propose a doctrine which may, I fear, appear wildly paradoxical and subversive. The doctrine in question is this: that it is undesirable to believe a proposition when there is no ground whatever for supposing it true." His proposal was "wildly paradoxical" because Russell understood that most of us do not use reason to form our beliefs. Most of what we believe is formed instead by an eclectic brew of formative influences, emotional hunches and selective concentration on evidence that supports our own prejudices. We are all victims of this "confirmation bias", but most of us like to think we'd have the good grace to change our minds if faced with incontrovertible evidence.

Enter Will Storr, whose first book Will Storr vs the Supernatural (2006) was a journalistic investigation of "the truth about ghosts". With The Heretics he has returned to similar ground and, as if taking on Russell's mantle, sets out to question individuals who persist in beliefs about the world that are, in his view, demonstrably false. "I like to write about these people," Storr confesses, "it is like being a tourist in another universe." At first glance the chapters of this book seem like a straightforward series of magazine pieces, with Storr in the role of objective journalist driven to uncover the truth, then persuade others of his arguments. We expect him to poke fun at some of the wackier believers – the UFO-spotters, yoga fundamentalists and past life regression fanatics have some rollicking arguments with the belligerents among them, and emerge crowned in the conclusion, a champion for science. But as you read on, something darker, more complex, and potentially more interesting begins to unfold. The other universes he tours are brought rather closer to home: the heretics "are magic makers", he tells us, and he feels "kinship" with them. This is not what we expect of the sceptical journalist. He confounds other expectations by being naturally shy, and by refusing to champion anything. He confides that he likes being a journalist because it frees him from the demands of social conversation, and that he is at ease in only two situations: with a voice recorder and a list of questions, or lying on his therapist's couch.

Storr can open chapters like a stage conjurer, and his prose has an easy, laconic style embracing Jon Ronson's taste for the fabulously weird and Louis Theroux's ability to put his subjects at ease. He is a funny and companionable guide (though occasionally he breaks his flow with excessively transcribed dialogue). Current neuropsychology is a theme, and the two chapters he spends exploring theories of mind, emotion and intuition are among the best.

As the book goes on, we understand it to be as much about the author's own therapy as it is about the subjects of his investigations. We hear how he dropped out of college, fought with his parents, stalked a former girlfriend, fell into drug abuse and petty crime. We see him anguished, contemplating suicide, and masturbating on a meditation retreat. For the most part this level of candour isn't distracting, but endearing. But by the end of the book I began to worry a little for him, too.

The book opens with a quote from the psychologist Jerome Bruner: "A self is probably the most impressive work of art we ever produce." Storr wants us to understand that it is not just "heretics" but all of us who are riddled with untested beliefs. In his search to understand how we construct our inner worlds he talks to chronic schizophrenics who have fought against the prevailing medical model of their disease, and attempted to view the particularities of the way they see the world more creatively. He spends time with a creationist, trying to understand how he can explain away the fossil record. He has a chilling encounter with a therapist whom he suspects of implanting false memories of demonic sexual abuse in the minds of vulnerable patients. At a "sceptics" conference he questions some self-righteous individuals with a passionate hatred of homoeopathy, only to discover that none of them has read the research that they refer to. One of the most pitiable groups he encounters are the sufferers of delusional parasitosis, who scratch their skins until they bleed because of a belief that they are infested by insects ("morgellons") despite the lack of any evidence to support them. But it is towards the end of the book, when he meets the right-wing ideologue Lord Monckton and the Holocaust denier David Irving, that he begins to excavate darker and more dangerous territory – the places where humanity's wilful self-delusion can lead. Going undercover with some neo-Nazis he finds himself standing in a gas chamber, listening to Irving point out how the chamber is a fake, a "typical Polish botch job". Shuddering with horror, Storr turns away from the group and begins to cross himself.

And so the fearless defender of reason finds himself warding off evil with a gesture from a religion that he professes to have disowned. This is a revelatory moment: he has demonstrated that we are all capable of prejudice, and each of us is an "enemy of science" in our own way. But what matters most to Storr is that we don't find ourselves enemies of compassion and understanding as well. We can believe what we like, as long as we don't hurt others with our beliefs. The book could have ended there, but there are two more chapters (on ESP and the arch-sceptic James Randi) and then an epilogue, none of which manages to build on Storr's moment of clarity in the gas chamber – the real climax of this uneven but thought-provoking book.

I began reading The Heretics thinking that Bertrand Russell, that old pipe-smoking pacifist, would have approved of it, but by the end I was doubtful. About the widespread adoption of sceptical rationality Russell wrote: "these propositions may seem mild, yet, if accepted, they would absolutely revolutionise human life." He wanted to see reason adopted more thoroughly, despite knowing that it would "diminish the incomes of clairvoyants, bookmakers, bishops, and others who live on … irrational hopes". But Storr doesn't want to do away with his heretics – their beliefs "glitter and pulse and enchant" while the cold rationalists look on glumly, touting "glitter-extinguishers". His enthusiasm for the straggling cavalcade of human beliefs is at odds with some of his other conclusions, but maybe it's churlish to expect consistency. "Bring on the psychics," he declares towards the end. "Bring on the alien abductees – bring on a hundred of them … Wrongness is a human right."

• Gavin Francis's Empire Antarctica is published by Chatto.