Combing over the newly released files on the assassination of John F Kennedy, as so many are now doing, may answer some important questions about an event that shocked the world and still fascinates it more than half a century on. But it will spark many more – and not only because some documents were held back at the last moment (what are they trying to hide?). Conspiracy theories are flourishing in democracies as never before.

They resist both counter-proof and persuasion because their appeal is primarily emotional. They posit an underlying order that is, on some level, reassuring – hence their proliferation after the sudden deaths of powerful, glamorous figures such as JFK or Diana, Princess of Wales. The system may be malign, but at least it is not random. If only it can be properly exposed, these evils may even be overcome. It may take mental acrobatics to fit the theory to the facts, but there is still a clarity lacking in the muddiness and opacity of real life. A theory of everything holds Freemasons or lizard-people responsible for all that goes wrong. A plot reassures us we are not at the mercy of the interaction of complex and often random forces. A delightful sense of superiority bolsters these stories – you have understood a truth that others are too naive to see, or too corrupt to admit. They often play to existing prejudices, notably antisemitism.

But these ideas can be more rational than their baroque surfaces suggest. Some conspiracy theories prove true. Others are underpinned by truths: JFK had many enemies; parts of the state were secretly smearing political enemies, overthrowing democratically elected governments elsewhere and assassinating leaders; there was a cover-up.

Social media and its echo chambers mean such theories are more visible and rapidly spread than ever. The consequences can be pernicious: ask the bereaved parents of Sandy Hook, hounded with death threats by those convinced the massacre was staged by actors. And while there have been periods when theorising has peaked before, the problem this time may be endemic.

Research has previously suggested that conspiracy theories are for losers – literally. Claims that the president is secretly a communist gain ground among Republicans when Democrats are in the White House; stories about Republican wrongdoing advance among out-of-power Democrats. But David Runciman, co-director of the Conspiracy and Democracy project at Cambridge University, warns that the issue now is not the volume of conspiratorial theories but their spread across the political spectrum, from left to right, and their exploitation by democratically elected politicians as well as authoritarian regimes. Donald Trump aggressively promoted the idea of elites manoeuvring against the masses. Such explanations divert blame to a scapegoat, confuse the issue or simply distract from more obvious sins. They are as useful in power as on the campaign trail.

Lies and distortions spread via social media undermine democracies; but they are more of a symptom than a cause. Even nonsensical populist fables have a kind of logic at their core. Research by YouGov for the Conspiracy and Democracy project suggests that people who are politically and socio-economically excluded are the most likely to believe conspiracy theories and that such beliefs have less sway in more equal societies.

It is increasingly obvious that democratic politics fails to live up to its promises and that the world is indeed controlled to a large extent by elites who defend or expand their riches and power. Wealth is at its most concentrated since the US Gilded Age. Oxfam has revealed that eight people own the same wealth as half the worldAddressing these deeper problems is the best way to root out conspiracy theories. Until then, they are likely to thrive – legitimate grounds of anger becoming diluted into a paranoid search for the missing link, as Professor Runciman puts it. You might almost think someone was trying to distract you.