In the early hours of April 20 1995, police knocked on the door of McArthur Wheeler and arrested him for holding-up two Pittsburgh banks the previous day. Wheeler could hardly have been surprised that the police were on to him: wearing no mask or disguise, he had ambled into the banks during business hours and brandished a gun in full view of security cameras. Nevertheless, he was astonished, protesting “but I wore the juice!” Wheeler had formed the erroneous belief that lemon juice rendered people invisible on video.

Wheeler is now a legend in psychology, since it was his regrettable escapade that inspired two psychologists, David Dunning and Justin Kruger, to figure out whether we have a good sense of our own strengths and weaknesses. Dunning and Kruger set tests of grammar, logic and even having a sense of humour to a group of undergraduates. Then they asked them how they stacked up to others in the group. Was their grasp of logic and grammar better or worse than average? Were they better able than other students to distinguish funny from unfunny jokes?

Most students thought that they were above-average logicians, grammarians and wits but the Dunning-Kruger effect is not mere overconfidence. The competent people in the study had a reasonable grasp of where they stood in the pecking order. The incompetent ones were blissfully unaware of their incompetence. The good students knew that they were good; the bad students had no clue that they were bad.

Perhaps because Dunning and Kruger opened their 1999 research paper with the story of McArthur Wheeler, the Dunning-Kruger effect has now become a popular insult in some corners of the internet. We chuckle at people who are far too stupid to know that they are stupid. Unfortunately, such mockery misses the subtlety and universality of the effect. All of us are incompetent in some areas. When we stray into them, the Dunning-Kruger effect may be lurking.

The fundamental problem is a person trying to diagnose his own incompetence is — almost by necessity — likely to be missing the skills needed to make that diagnosis. Not knowing much grammar means you’re poorly placed to diagnose your ignorance of grammar.

There is, of course, a cure for the curse of Dunning-Kruger: asking for advice or criticism. On the question of whether lemon juice is an invisibility potion as well as an invisible ink, McArthur Wheeler could have benefited from a second opinion. Doing so, alas, would have required him to doubt his own reasoning on the matter; it would also have required him to identify a bright-enough advisor. And all of us — especially high-status people — face the problem that when we are sorely mistaken, our friends and colleagues are often too polite to tell us. Still: two heads are better than one.

In a new book, Head in the Cloud (US) (UK), William Poundstone argues for a fresh defence against Dunning-Kruger catastrophes: trivia. Poundstone believes that a broad base of knowledge helps to clue us in to the times when we are stumbling towards a humbling; if we know a little about a lot, we have more opportunities to catch ourselves in the middle of a Dunning-Kruger moment.

Poundstone’s own research suggests that there’s a correlation between income and general knowledge, over and above what might be expected from education levels. One of many plausible explanations: people with a good grasp of general trivia are people who are paying attention to the world.

This is highly speculative stuff, but thought-provoking. Poundstone is pushing against the tide: educational fashion, as well as common sense, suggests that in the age of the smartphone it is better to focus on critical thinking than on rote memorisation. But he may have a point; any particular fact can be looked up, but without a knowledge base who knows where to start?

Recently, psychologist Sarah Tauber and four fellow researchers posed a long series of trivia questions to hundreds of young people (their average age was 20). An example: “What is the name of the large hairy spider that lives near bananas?” Despite a generous marking system (for example, “teranchula” was considered correct), performance was unimpressive. Fewer than half the subjects knew which country’s capital was Baghdad, or what the spear-like object was that was thrown around in athletics contests. Only 43 per cent knew that the hairy spider was a tarantula, however spelt.

As for more challenging pieces of trivia, the performance was astonishingly bad. Name Flash Gordon’s girlfriend, or the author of The Brothers Karamazov, or the first man to run a four-minute mile, or the mountain range separating Europe from Asia? Out of hundreds of participants, nobody knew. Nobody. There were 50 such questions, questions to which not a single person could venture a suitable guess. And these 20-year-olds were undergraduates, so presumably reasonably smart and ostensibly well educated.

It’s not that young people today are stupid. They’re the most educated generation in history, and their intelligence is higher, at least as measured by IQ tests. It’s just that there’s a lot they don’t know, and (as per Dunning-Kruger) a lot they don’t know they don’t know. I’m not sure that is a problem but it might be. As Poundstone points out, one thing you cannot Google is what you should be googling.

Written for and first published in the Financial Times.

It helps any new book to pick up some advance orders, so if you like my writing please consider pre-ordering my new book, “Messy” .(US) (UK) More to follow soon…