The death of Jeffrey Epstein arrived at the perfect time for America’s conspiracy industry. The Trump-Russia story had just been rendered barren by Robert Mueller’s stubborn refusal to deliver the goods. Laurence Tribe, a law professor who acquired a huge Twitter following on the back of his overheated commentary on the Russia investigation, jumped on the Epstein news, tweeting, “You don’t have to be a conspiracy theorist to see an evil cover-up to protect lots of powerful men here.” Well, I’m not a law professor but I think you probably do, don’t you?

Personally, I’m a cock-up theorist. I instinctively disbelieve almost any explanation of events that proposes organised malevolence over accident and mistake. I think this is the right bias to have, if only because most people just aren’t organised enough to pull off dastardly secret schemes. The rewards of a high-level conspiracy rarely outweigh the risks, or the effort – just think of the admin.

The case of Epstein tests my conspiracy scepticism to the limit, however. Somehow the world’s most high-profile felon – a man who consorted with the planet’s most powerful and wealthy figures, and who was charged with sex trafficking – contrived to kill himself while in a federal jail, under the noses of guards with strict instructions to stop him killing himself. I’ll admit it doesn’t look good, and yet, even here, I still feel it was more likely to have been a cock-up than not. I may be wrong, though. After all, anyone who applies the cock-up rule uniformly is going to miss some big, stinking, dishonest-to-God conspiracies.

In 1994 the journalist Dan Baum was researching a book about America’s war on drugs. He wanted to know more about why Richard Nixon initiated it, so he tracked down John Ehrlichman, a former Nixon adviser who spent time in jail for his role in Watergate. “You want to know what this was really all about?” said Ehrlichman, after waving away Baum’s earnest policy questions. “We knew we couldn’t make it illegal to be either against the war or black, but by getting the public to associate hippies with marijuana and blacks with heroin, and then criminalising both heavily, we could disrupt those communities.” (When Baum met him, Ehrlichman was working for a law firm in Atlanta, on minority recruitment.) “Did we know we were lying about drugs? Of course we did.”

Watergate itself, of course, was a prime example of a real conspiracy – and there are others, closer to home. In 2011 the retired judge Oliver Popplewell warned the families of Hillsborough victims not to “harbour conspiracy theories”. It wasn’t the first time that the families had heard this, and it would be another five years before the world accepted what they had believed all along: that the police did indeed conspire to blame the victims.

Not all conspiracy theories are wrong. Actually, I was being unfair to Professor Tribe, who was making, or implying, an interesting distinction. You can believe in a conspiracy theory without being a conspiracy theorist – that is, someone who adopts conspiracy as their default explanation for events. The French psychologist Serge Moscovici identified this as “the conspiracy mentality” and empirical research has shown it to be a real thing: endorsement of conspiracy theories is associated with belief in other conspiracy theories. While most people assume cock-up unless proven otherwise, there are others – perhaps a growing number of others – who assume the opposite. Why is that?

One of the main characteristics of conspiracy believers, according to multiple studies, is that they are less educated than non-believers. That doesn’t mean conspiracy theorists are stupid. In fact, they are good at performing one of the core functions of an intelligent mind: they discern patterns in complex data. The trouble is, their pattern-detection function is turned up to 11 – which, like feedback through an amp, generates a distorted signal. Researchers think that may have something to do with a lack of the analytical skills needed to parse complicated information. But there’s another driver of conspiracism too, which may be more fundamental.

Adam Galinsky and Jennifer Whitson, then of the University of Texas, asked people to think about a time when they felt threatened, and in which they either had or lacked control. The researchers then showed them images that featured random collections of black and white dots, and also told them stories about a bad event befalling someone. The people who had been primed to feel out of control were more likely to see patterns in the randomness – and to believe that the story’s protagonist was the victim of people ganging up on them.

Believing in a conspiracy is a way of imposing some kind of order on a world that can seem unsettlingly chaotic and opaque. It’s an instinct to which most of us are susceptible, at some level. Stories such as The Da Vinci Code, The Matrix and The X-Files are enduringly popular because they resonate with the way people feel about the world of giant corporations, omnipresent technology and strange weather. We are tempted to believe that someone, somewhere out there is manipulating the conditions of our lives to nefarious ends. If our misfortunes result from the actions of a few bad actors rather than, say, the unfathomable workings of the global economy, that makes them easier to grasp.

This is the paradox of conspiracism: one way to feel a little more in control of the world is to believe that others are exerting control over us. It can be hard to accept the truth: that in our vast and complex societies, control over anything by anyone is harder to achieve than ever. Then again, maybe that’s just what they want us to think.