Why are the New Atheists such jerks? Case in point: Richard Dawkins’ continuing pursuit of Ahmed Mohamed, the Texas 14-year-old humiliated in school after authorities mistook his homemade clock for a bomb.

The other day, The God Delusion author called Ahmed a hoaxer and responded to suggestions “he was only a kid” by linking to a report about a juvenile Islamic State (Isis) fighter. “And how old is this ‘kid’?” Dawkins asked.

Dawkins has been after the teenager for some time. When the story of Ahmed’s arrest and interrogation in handcuffs first broke, Dawkins questioned the boy’s motives, before linking to a video suggesting Ahmed wasn’t quite the inventor as he claimed.

“Assembling clock from bought components is fine,” tweeted Dawkins to his 1.3m followers. “Taking clock out of its case to make it look as if he built it is not fine. Which is true?”

The intervention exemplified everything toxic about Dawkins’ online persona. It’s not just the unedifying spectacle of an internationally famous biologist seeking to discredit a teenager’s science project, like a 9/11 truther obsessing about jet fuel. It’s also Dawkins’ disgraceful juxtaposition between Ahmed and an Isis supporter in Syria.

“How COULD you think I was likening a hoaxer to a killer?” he later posted. “I just meant ‘Only a kid’ is not a knockdown defence. Remember poor James Bulger?”

Except, of course, Dawkins hadn’t compared Ahmed to Bulger, who was murdered in 1993 by two 10-year-old boys. He’d linked a youth falsely accused of terrorism on the basis of his religion to Isis, precisely the kind of smear that any FOX news demagogue might make.

Then there’s the author, philosopher, and neuroscientist Sam Harris – another New Atheist luminary.

A few days before Dawkins relaunched his investigation into clock-gate, Harris explained on a podcast that Republican hopeful Ben Carson understood the Middle East better than Noam Chomsky. The same Ben Carson who thinks there’s a scientific consensus that aliens built the pyramids (even though Carson knows they were actually built by God as a granary for the biblical Jacob).

Why does Harris prefer Carson’s know-nothing bluster on foreign policy to the opinions of Chomsky, one of the most influential scholars in the world? Because, you see, Carson “understands that jihadists are the enemy”. That’s also why Harris defends Ted Cruz’s proposal to screen Syrian immigrants according to their religious views, since “some percentage of Muslims will be jihadists inevitably”.

In 2011, the leftwing atheist writer PZ Myers took offence with an article I wrote calling out the anti-Muslim bigotry of the most prominent New Atheists. Back then, Myers denied that Harris was rightwing and complained:

Sparrow condemns us because we haven’t thrown Hitchens from our ranks, and that we’re supposed to “speak out against the Islamophobia that’s self-evidently rife in the atheist movement,” a perfectly lovely demand that is offensive in its assumptions; shall he also tell me that I must stop beating my wife? There is racist Islamophobia scattered about within the New Atheist movement ... But the outliers are not the movement.

These days, Myers (to his credit) devotes considerable time to denouncing Harris. My favourite of his recent interventions includes the line: “Sam Harris [is] full of paranoid, racist shit.”

But the question remains: how did we get here?

Dawkins and Harris are still, by far and away, the most recognisable frontmen for the New Atheist show. So how did a movement ostensibly full of progressives end up so identified with writers who sound less and less like incarnations of pure reason and more and more like your Islamophobic uncle after he chugs his sixth pint?

The novelty of New Atheism comes from its contrast with an older atheism, associated throughout the 19th and 20th centuries with the left in general and socialism in particular. That’s why, for a certain generation of right-wingers, the epithet “commie” invariably follows “godless”.

By the 2000s, the old left had disintegrated, both as a movement and a set of ideas, even as some of its doctrines became entirely mainstream. Secularism was one of them. In 1901, it took considerable courage to proclaim your atheism in an English-speaking country; a century later, non-belief had become (within the intelligentsia, at least) largely unexceptional.

That was part of what made the New Atheists new. An earlier generation of atheists were brash and offensive but their provocations were generally directed at a church that still possessed considerable institutional power. The New Atheists were, by contrast, insiders rather than outsiders, writing and speaking in societies where manifestations of fervent religiosity largely occurred on the cultural fringes rather than the intellectual centres.

(Even in America, something of an anomaly on these matters, religious presidential candidates direct their evangelical huckstering at Smallville, USA and not the sophisticates of the big cities).

As a philosophical tendency, the New Atheists were popularisers rather than innovators, using advances in biology and neuroscience to illustrate pretty well-worn arguments against religion. Indeed, in some crucial ways, they represent an intellectual step backward from a left that had recognised atheism as necessary but scarcely sufficient.

As early as 1842, Marx dismissed those who trumpeted their disbelief to children as “assuring everyone who is ready to listen to them that they are not afraid of the bogeyman”. For him, intellectual disproofs of God were trivial; what mattered was building a world that didn’t give rise to mystification of any kind.

That is, if you investigate the material basis of religious belief, you immediately confront a phenomenon that operates on many different levels. In particular circumstances and particular settings a faith may function as a guide to morality, or an aesthetic, or a social network, or a collection of cultural practices, or a political identity, or a historical tradition, or some combination of any or all of those things.

You don’t have to be a believer to see that religion genuinely offers something to its adherents (often when nothing else is available) and that what it provides is neither inconsequential nor silly.

By contrast, the New Atheists engage with religion purely as a set of ideas, a kind of cosmic rulebook for believers. On that basis, it’s easy to point out inconsistencies or contradictions in the various holy texts and mock the faithful for their gullibility.

But what happens then? You’re left with no explanation for their devotion other than a susceptibility to fraud. To borrow Dawkins’ title, if God is nothing but an intellectual delusion then the billions of believers are, well, deluded; a collection of feeble saps in need of enlightenment from their intellectual superiors.

That’s the basis for the dickishness that so many people now associate from the New Atheism, a movement too often exemplified by privileged know-it-alls telling the poor that they’re idiots. But that’s only part of it. For, of course, the privileged know-it-alls are usually white and those they lampoon the most are invariably Muslim.

For the extraordinary contemporary popularity of the New Atheism also relates to something else that happened at the dawn of the new century – namely, the terrorist attacks on 2001. It’s 9/11, more than anything else, that divides the old atheism from the new.

The best illustration is Christopher Hitchens, a writer who built his stratospheric literary career by transitioning between the two atheist traditions. As a young man, Hitchens was a Trotskyist and for many years he remained a leftwing polemicist. During that time, his atheism attracted no particular attention: it went almost without saying that a prominent representative of the British left didn’t believe in God.

By 2001, Hitchens was already beginning his shift to the right. 9/11 provided the catalyst for a complete break. He signaled the shift with an extended polemic against – you guessed it! – Noam Chomsky, the man Sam Harris distrusts so much.

Chomsky insisted (then as now) that bin Laden arose from a particular context and history, that al Qaeda wasn’t merely the result of inexplicable Muslim rage. Hitchens, like Harris, would have none of it. It was actually Chomsky who had “lost or is losing the qualities that made him a great moral and political tutor”.

Hitchens, for his part, wrote “that the forces represented by Al Qaeda and the Taliban are fairly easy to comprehend, but not very easy to coexist with”. He deployed a vulgar critique of religion, along lines that are now so drearily familiar.

The problems in the Middle East stemmed, not from imperial meddling in an oil-rich region but from Islam itself, a faith that resulted from (and then fostered) delusional thinking. On that basis, Hitchens was increasingly able to ally himself with the worst elements of the American right while insisting he remained a progressive.

You can see how the argument works. If belief in God stems from intellectual inadequacy, then all believers are feebleminded – and the most devout are the most feebleminded of all. All religions are bad but some religions – especially those in the Middle East, by sheer coincidence! – are worse than others.

In the name of enlightened atheism, you thus arrive at an old-fashioned imperialism: the people we just happen to be bombing are simple-minded savages, impervious to reason and civilisation. That was the secret of Hitchens’ success: he provided a liberal rationale for the “war on terror”.

You can proclaim you’re an atheist, a freethinker, a devotee of the enlightenment – and yet somehow still end up backing rightwing Christians like George W Bush and Ben Carson in their campaigns against the Muslim hordes.

Which is why it’s not enough to denounce Dawkins and Harris. If we’re to save the good name of atheism, we need to popularise a fundamentally different approach, one that seeks to understand religion rather than simply sneering at it.

As my colleague Jason Wilson argues, denunciations of other people’s “stupidity” are a particular temptation of our age. By way of contrast, he puts the case for solidarity, writing that:

Solidarity requires listening: to stories of the structural deformation of individual lives; to the ways that popular culture makes people feel like they are living against the grain; to analyses that have not yet and may never become wholly coherent, or even depart from common sense.

That doesn’t entail abandoning a critique of religion. But it does mean adopting a certain humility when coming to terms with why ordinary people believe the things they do.

In a different world, religion might not be necessary. But we’re not in that world yet. In the struggle for social change, the religious will play just as important role as anyone else. If you don’t believe in God, that’s great. But you’re not helping by being a jerk about it.