Nothing makes the contented turn of the century feel further away than the indulgence with which the old world treated its cranks. Their prime purpose was to be entertaining freaks for the allegedly sane majority to laugh at. The BBC ran shows where Louis Theroux met religious zealots and white nationalists. As they watched, broadcasters and the audience had an unspoken pact that made sense 20 years ago but is meaningless today: however dangerous these people might be to those close to them, they could do no real harm.

The ironic documentaries of the 1990s now seem as remote as medieval frescoes. If producers wanted to commission a successor series, they would have to take their cameras to the White House, Kremlin, the office of the leader of the opposition in Westminster, the Sándor Palace in Budapest and Chancellery in Warsaw. They would have to ask how a Russia that has turned paranoid delusion into an instrument of foreign policy became the dominant power in the Middle East and a corrupting force in the west. They would have to acknowledge that conspiracy theorists rule nuclear-armed states and that the fake news and the loud-mouthed bombast of men once dismissed as clowns on Have I Got News for You pushed Britain out of the EU. It’s the clowns who are laughing now and the cranks who rule the world.

Beyond the well-explored reasons for the rise of demagogic movements lies an embarrassing failure. For once you have examined the effects of the crash, the stagnation of living standards, austerity, the mass movement of migrants, racism, globalisation, the influence of the web and Russian money, you still have to ask why mainstream society did not take demagogues seriously until it was too late.

Until the moment Trump was elected American politicians and journalists refused to believe that a man like him could possibly be their president. No senior conservative politician in office said the only way to preserve American democracy was for moderate Republicans to hold their noses and vote for Hillary Clinton. That they are cowards has been proved by their collusion since Trump came to power. But they seem as much fools as villains when you remember they thought he was doomed to lose. David Cameron called the Brexit vote with an unshakeable conviction that he would win and carry on in Downing Street. Centrist MPs turned Labour from a social democratic party into a playground for post-communist tyrannophiles when they put Corbyn on the ballot paper to “have a debate”, and then found they couldn’t debate him because they had never once tried to understand the strengths and weaknesses of the new far left.

All of us can mistake the familiar for the permanent and find change inexplicable. But there was a deeper fault. The comforting idea that conspiracy theorists didn’t matter was not confined to documentary-makers. When confronted with half-mad ideas that global warming was a lie or that the CIA destroyed the twin towers, intelligent people reassured themselves that these were delusions shared by a few people who needed the simple explanations global conspiracies gave them to make sense of life’s chaos.

Richard J Hofstadter’s classic essay of 1964, The Paranoid Style in American Politics, is still read today. But its conclusion was surprisingly optimistic. Although conspiracy theories were “a persistent psychic phenomenon”, they affected only a “modest minority of the population”. That Hofstadter could be so nonchalant when Nazi Germany was fresh in the memory and communists still ruled the Soviet Union and China shows that western parochialism is not a novel vice.

Forensic police officers examine a vehicle believed to belong to the poisoned Russian double agent Sergei Skripal. Photograph: Rufus Cox/Getty Images

Grand historical objections aside, it is obvious to anyone who has argued with conspiracy theorists that their ideas are anything but simple. When you imagine global warming is a hoax you need hundreds of pieces of scaffolding to support your fantasy. You must believe that about 95% of climate scientists are lying and that none ever had or would ever have a crisis of conscience and confess “the truth”. The complexity of the delusion explains why conspiracy theory so often ends in fascism. For such a huge and devilish con to be pulled, true believers have to invoke a group with supernatural powers to arrange and conceal: the Jews or the “Zionists”, as they say today.

They also have to explain motive. This problem is haunting the Corbynite left and Faragist right as they struggle to explain why the attempted murder of a Russian double agent and his daughter is not the work of a Russian state which murders its critics as a matter of routine. They blame the Americans, the May administration, which doesn’t have the competence to conspire its way out of a paper bag and, inevitably, the Jews. Corbyn eggs them on by saying that “Russian mafia-like groups” may be behind it. But why would the mafia want to do it?

It’s easy to sink into despair now the cranks and creeps aren’t a “modest minority of the population”. When Hitler and Stalin controlled mainland Europe, Stefan Zweig looked back with nostalgia to a time 30 years before when “there was as little belief in the possibility of such barbaric declines as wars between the peoples of Europe as there was in witches and ghosts. Our fathers were comfortably saturated with confidence in the unfailing and binding power of tolerance and conciliation. They honestly believed that the divergences and boundaries between nations and sects would gradually melt away.”

Zweig committed suicide in 1942. The alternative to despair is to fight and find a pleasure in fighting. Many, myself included, are enjoying the grim satisfaction of engaging in arguments that have a significance the trivial pursuits of the millennium could never claim. No one can wake up now and say there are no good causes left. They need only look around to see dozens of deserving targets that demand to be hit.

• Nick Cohen is an Observer columnist.