If the Illuminati is real, it’s got to be the least secret secret society in the universe. It’s so bad at keeping itself hidden that its existence is proclaimed all over the internet by people whose investigative toolkit consists entirely of Google and a lively imagination.

The most recent would-be whistleblower, however, is far from your usual ex-sports commentator. Paul Hellyer, a former Canadian minister of defence, has blamed the Illuminati for suppressing technology brought to Earth by aliens that could end our reliance on fossil fuels.

What is the Illuminati? You asked Google – here’s the answer | Martha Gill Read more

Why the possessors of such fantastic kit should prefer to cash in on the extraction of still abundant oil rather than on their incredible, exclusive alternative is mysterious. But since the whole point about secret all-powerful elites is that they are mysterious, maybe that’s to be expected. Perhaps the Illuminati is like that other great mystery, quantum theory: if you think you understand it, you don’t.

Mockery is easy, but it’s also reassuring. It’s good to know that we’re much more sensible and rational than these clearly deluded conspiracy theorists. The problem is that they differ from the rest of us only in degree, not kind.

The reasons why people believe in secret, controlling elites are rooted in basic human nature. We are constantly on the lookout for both patterns and agency. Pattern-seeking is essential for our survival, and the penalties for seeing patterns where none exist are lighter than those for missing patterns that really are there. If our ancestors had failed to notice that crops left to dry tended to die, they too would have expired through starvation. But if they thought they had noticed that sacrificing a goat increased the likelihood of rain, then at worst they wasted the odd bit of meat.

The assumption of agency is also extremely helpful. We cannot begin to understand the actions of others unless we attribute motives to their actions. But adopting what the US philosopher Daniel Dennett calls the “intentional stance” can be helpful even when we know that there is no conscious intention at all. Thinking of plants as “wanting” sunlight or “trying” to flower, for example, is an easy way of understanding their behaviour.

When these basic human cognitive mechanisms create problems we label them as pathologies. Pareidolia, for instance, is seeing patterns in random data, such as the face of Jesus in a cream cracker or the date of the apocalypse in Donald Trump’s social security number. However, from a strictly rational point of view, these mechanisms are always defective. The difference between a “normal” person and one with pareidolia is simply whether the over-sensitivity to pattern causes problems functioning. Similarly, hyperactive agency detection is the human condition, not a medical one.

Furthermore, the causes of our overuse of these basic cognitive mechanisms are often completely understandable desires rather than pathological defects. The world is confusing and complex. Almost all of us try to tidy it up, which is why there are disciplines such as social science, economics and international relations. All require the time, intelligence or education that most of us lack. Little wonder that we often resort to quicker, dirtier ways of making the world comprehensible.

The world is confusing and complex. When we dig for the truth, we flirt with madness

Rather than just dismissing Hellyer and his ilk, we would do better to see how much of our thinking displays the same weaknesses. In many circles, it is a sign of intelligence, not eccentricity, to attribute ultimate power to “a secret cabal that’s actually running the world”, as Hellyer put it. As long, that is, as the cabal is the global financial elite, the military–industrial complex, big pharma or agribusiness.

I’m not saying that these ideas are on the same level of nuttiness as the Illuminati. Indeed, it’s the differences that blind us to the similarities. Because these lesser conspiracy theories are grounded in evident truths, people easily fail to notice when they slide from seeing real, limited power to imagined, total power. The difference between vested interests that exercise influence all over the place and those that exercise control literally everywhere is in some ways small, in other ways critical.

The wrong moral to draw from this would be that anyone who sees hidden power being influenced is crazy. Rather, we should see the Hellyers of this world as the price we pay for being willing to question the manifest order and to expose the secretive interest groups who seek to manipulate the world for their own benefit. When we dig for the truth, we flirt with madness. But in a world where hidden power is all too real, it’s the only sane thing to do.

• Julian Baggini is a British philosopher