‘I don’t understand what you think that I’m saying.’

(Marital argument, overheard at a local diner.)

Following a mass shooting, the gunman’s next-door neighbours are stunned, and tell reporters that he was a good, kind man. Meanwhile, former classmates and co-workers describe him as a ticking time bomb. Pundits attribute Donald Trump’s latest Twitter tirade to unbridled narcissism, early dementia, a bullying father, Machiavellian shrewdness – or a man with a heartfelt mission to make America great again… Show us any human behaviour, and we’ll drum up half a dozen seemingly common-sense explanations. The underlying assumption: we can know with a reasonable degree of accuracy what is going on in another’s mind. Labelled by psychologists as theory of mind (abbreviated as ToM), this ability to understand that others have separate minds containing potentially different beliefs, desires and intentions is often said to be one of our pre-eminent cognitive skills distinguishing us from other creatures.

That we have a folk psychology theory of other minds isn’t surprising. By nature, we are character analysts, behavioural policemen, admirers and haters. We embrace like minds, and go to war against contrarians. Mind-reading is our social glue, guiding virtually all of our daily interpersonal interactions. When trying to decide whether or not a potential gun owner is prone to violence, a mental patient is suicidal, or a presidential candidate is truthful, we are at the mercy of our thoughts about others.

The fate of democracy depends on our ability to grasp and accept differing mindsets – yet the seemingly near-universal absence of reasonable public discourse suggests that this rarely occurs. We accuse those with conflicting opinions of having character defects, subliminal prejudices, faulty education, cultural brainwashing and a myriad of other ‘if only they knew better’ flaws of reasoning. But there’s a more basic and frightening possibility. What if we really aren’t capable of a sophisticated reading of other minds?

For starters, let’s pretend the impossible – that we can step outside our minds and see how ToM might work. A psychologist presents a child with two hand puppets – Sally (who has a basket) and Anne (who has a box). Sally puts a marble in her basket, and leaves the room. While Sally is away, Anne takes the marble from the basket, and hides it in her box. Finally, Sally returns to the room, and the child is asked where Sally will look for her marble. By around age four, most children recognise that Sally will look in her basket (where she last had the marble), not in Anne’s box. Absent neurodevelopmental abnormalities such as autism, this universal ability of young children to pass various versions of the Sally-Anne test is frequently cited by cognitive scientists as unequivocal evidence that we can know the minds of others.

To offer further insight, in recent years, neuroscientists have come up with some tempting theories of how our brain might accomplish this feat. The first promising neural mechanism was described in 1992 when the Italian neuroscientist Giacomo Rizzolatti and colleagues discovered that when rhesus macaque monkeys reach for food such as peanuts, individual cells fire in the prefrontal motor cortex. The same cells also fire when the monkeys observe a researcher reaching out for a peanut – as long as the monkey believes that the gesture is intentional and that the experimenter planned to eat the peanut. Since the same cells fire both when initiating and observing an action, they have been labelled ‘mirror neurons’; collectively, the network is called the ‘mirror neuron system’.

Because the macaques could tell whether a gesture was made in order to eat a peanut or to play with it, researchers postulated that the mirror neuron system could detect intention – and that the monkeys possess a theory of mind. During the decade following their discovery, in fact, mirror neurons were frequently touted as the neural basis for empathy, complex social interactions, evolution of language and cultural advances characteristic of the modern human. The behavioural neurologist V S Ramachandran even went so far as to assert that mirror neurons would ‘do for psychology what DNA did for biology … With knowledge of these neurons, you have the basis for understanding a host of very enigmatic aspects of the human mind: “mind-reading” empathy, imitation learning, and even the evolution of language.’

Ultimately, cooler heads prevailed, and skeptics downgraded the hyperbolic attributions. Marco Iacoboni, a neuroscientist at the University of California in Los Angeles and a pioneer in the mirror neuron work, said the system operated at the basic level of recognising simple intentions and actions – much like what we might do in a high-stakes poker game. You are about to make a bet when you notice that the player to your left is getting ready to push forward a stack of his chips. He might be making this gesture to distract you from another aspect of the game. Perhaps he is trying to distract you from his undercover partner, a player to the right. He might be trying to create a fake ‘tell’ to use against you in the future. A diversity of mental states can generate the same motor action. Understanding that your opponent will soon push his chips forward tells you nothing about the purpose behind the motion.

That hardly stopped scientists from trying to prove a theory of mind. With the collapse of the mirror neuron theory, other brain regions emerged as replacement candidates. In her wildly popular 2009 TED lecture, the cognitive scientist Rebecca Saxe at MIT argues that the right temporoparietal junction (rTPJ) – the region of brain just behind your right ear – ‘is almost completely specialised. It does almost nothing else except think about other people’s thoughts. Differences in this brain region can explain differences in adults in how we think about and judge other people.’

But we also know that the rTPJ coordinates incoming sensory inputs to create a stable physical sense of self in the environment. Transcranial magnetic stimulation can alter rTPJ function to produce the classic out-of-body experiences. Damaging the region via stroke or brain tumour can result in disordered self-awareness, even lack of recognition of paralysis. Despite this, according to Jean Decety, a cognitive scientist at the University of Chicago, a properly operating rTPJ is also necessary for us to distinguish ourselves from others.

It’s a strange loop: we are asking the same area of brain to both generate a coherent sense of self, and simultaneously step outside this frame of reference to get a fresh, unbiased perspective on another’s thoughts. Talk about running uphill against basic physiology.

Despite the inadequacy of these leading neuroscience explanations of ToM, it remains hard to shake the belief that we can step inside another’s mind. Saxe begins her TED lecture with the question: ‘How is it so easy to know other minds?’ To illustrate her point, she shows two photos. The first is a mother gazing at her young child; the second is of a teenager jumping off a high cliff into the ocean below. ‘You need almost no information, one snapshot of a stranger, to guess what this woman is thinking, or what this man is.’

I look at the mother, and see a combination of love and awe. But with a moment’s reflection, I realise that I have gathered together some general assumptions about what humans have in common, and dropped them into her mind. I have no way of knowing if she is also worrying that her husband might feel neglected by her single-minded devotion to her child, wondering when to enrol her child in preschool, or trying to cement into memory the feeling of unconditional love that she anticipates will be challenged when her newborn morphs into a rebellious teenager. By drawing on innate and acquired beliefs about human nature, I can imagine her mind at the most universal and generic, but not at the particular.

The photo of the boy diving off the cliff raises further questions. Since I’m unaware of any neuroscience literature on the mental states of cliff-divers, let me substitute a study on the world’s most famous free-soloing mountain climber, Alex Honnold. Just watch Honnold climb 3,000 feet (900 metres) straight up the vertical face of a Yosemite peak without the use of any safety equipment – no ropes, nets or harnesses. Ask yourself: is Honnold experiencing a great degree of anxiety and fear when he looks down at the Yosemite floor thousands of feet below – or a moderate amount? Or none? Also ask yourself how sure you are of your answer, and how you’d know if you are right.

In 2016, the neuroscientist Jane Joseph at the Medical University of South Carolina compared Honnold’s brain to that of another veteran mountain climber. While in an fMRI scanner, both were shown a succession of 200 supposedly high-anxiety-generating photos – gruesome burnt and disfigured corpses, mangled accident victims, as well as several high-risk mountain-climbing routes. The climber serving as a control demonstrated high-level activation of his amygdala – the area of brain that typically fires when one is fearful, frightened or anxious. By contrast, Joseph told Nautilus magazine, Honnold’s amygdala remained entirely silent. When asked about the photos, Honnold was puzzled. ‘I can’t say for sure, but I was like, whatever,’ he said. Even the ‘gruesome burning children and stuff’ photographs seemed to him dated and jaded. ‘It’s like looking through a curio museum.’

Trying to imagine a mental state you’ve never had is like trying to conjure what an orgasm feels like before you’ve ever had one

Joseph believes that Honnold’s fMRI reflects the absence of the normal primary threat response, as though his fear switch has been turned off. Even so, Honnold does not consider himself fearless. He recalls incidents both related and unrelated to mountain-climbing that he calls scary.

And now we run into a second problem – the superimposition of language on to mental states. Honnold is quite conscientious, and makes meticulous surveys of his climbing routes. He readily acknowledges that a fall means death, and describes this possibility as scary. Whether this represents a cognitive understanding of danger or a felt emotion is impossible to say. Given his non-firing amygdala, Alex’s ‘scary’ is unlikely to be similar to the type of fear other mortals get when standing near the ledge of a high-rise window, let alone a high cliff. Wondering what Honnold might experience while solo climbing reminds me of the philosopher Thomas Nagel’s unanswerable question: ‘What is it like to be a bat?’

This is not to say that we have no idea of what goes on in another’s mind. The brain is a superb pattern-recogniser; we routinely correctly anticipate that others will feel grief at a funeral, joy at a child’s first birthday party, and anger when cut off on the freeway. We are right often enough to trust our belief that others generally will feel as we do. Listen to the TED audience wince when shown the photo of the cliff diver as though personally sensing the fear he must be experiencing. And yet, if the cliff-diver has the same non-firing amygdala as Honnold, such impressions will be completely off-base. The insurmountable problem is that we are up against the limits of trying to imagine a mental state that we’ve never had. (Which is no different than trying to conjure what an orgasm feels like before you’ve had one.)

Perhaps I’m dead wrong and my theoretical objections don’t do ToM justice. Maybe there is compelling daily life evidence for ToM’s central claim that we can know the beliefs, desires and intentions of another.

Let’s start with the easiest way to study ToM experimentally – lie detection. If we are good at mind-reading, surely we should be superb lie detectors. But a 2006 review in the Journal of Personality and Social Psychology showed that volunteer subjects were barely better than chance at detecting when an actor was lying or telling the truth (54 per cent). A decade later, despite various efforts to improve lie-detection performance, the Monitor on Psychology reported that ‘people’s ability to detect lies is no more accurate than chance, or flipping a coin. This finding holds across all types of people – students, psychologists, judges, job interviewers and law-enforcement personnel.’

If we’re not so good at lie detection, perhaps we can do better at predicting violent behaviour. In 1984, The American Journal of Psychiatry reported that psychiatrists and psychologists were vastly overrated as predictors of violence. Even in the best of circumstances – with lengthy multidisciplinary evaluations of persons who had already manifested their violent proclivities on several occasions – psychiatrists and psychologists seemed to be wrong at least twice as often as they were right when they predicted violence. Nevertheless, the article suggested that new methodologies might improve prediction rates.

No such luck. Thirty years later, a review article in The British Medical Journal concluded that: ‘Even after 30 years of development, the view that violence, sexual or criminal risk can be predicted in most cases is not evidence-based.’ Despite being the co-developer of a widely used evaluation tool for violence risk-assessment, the psychologist Stephen D Hart at Simon Fraser University in Canada is equally pessimistic. ‘There is no instrument that is specifically useful or validated for identifying potential school shooters or mass murderers. There are many things in life where we have an inadequate evidence base, and this is one of them.’

Suicide prediction? Same story. According to two recent meta-analyses: ‘There has been no improvement in the accuracy of suicide risk-assessment over the last 40 years.’ The UK National Institute for Health and Care Excellence has advised that ‘assessment tools and scales designed to give a crude indication of the level of risk of suicide’ should not be used.

All good theories are predictive. Sooner or later, they need supporting evidence. If experts cannot tell us who will be violent, or commit suicide, or is lying, isn’t it time for us to reconsider whether there are real and practical limits to our belief in ToM?

No wonder Facebook has introduced its own AI system to detect those at increased risk of suicide

Earlier, I brought up the mirror neuron controversy to emphasise that there are a number of low-level brain processes that might appear like higher-level functions – but are not. I suspect that the Sally-Anne and other ToM tests are analogous examples. Yes, we know that other people have minds, desires and intentions that are potentially different to our own. But putting ourselves in another’s situation is not even remotely comparable to actually feeling and thinking as another. I might be able to fit into Honnold’s climbing shoes, but I cannot crawl inside his mind.

As I write this essay, I am reluctant to entirely accept the evidence that I’ve just presented. I cannot shake the gut feeling that there’s more to lie detection than is revealed in the studies. On the other hand, as an inveterate poker player, I admit to being shaky at detecting bluffs, so try to base my decisions upon player betting patterns. I’m not alone. Given the predictive failures of ToM, psychologists are increasingly looking at big data rather than individual minds.

A research team led by Stephan Ludwig at the University of Westminster in London developed automated text-mining software that analysed more than 8,000 emails bidding for awards based on a company’s performance. They compared the program’s ability to detect fibs in the bids against an independent investigation by the company’s account managers. The program far surpassed the account managers, achieving a 70 per cent degree of accuracy. The researchers hope that their technique will eventually be able to detect deception in everything from visa applications to dating profiles.

Scientists at Vanderbilt University Medical Center in Tennessee gathered data on more than 5,000 patients with physical signs of self-harm or suicidal ideation. By gathering up readily available impersonal healthcare data such as age, gender, zip codes, medications and prior diagnoses, but without directly interviewing the patients, there was 80-90 per cent accuracy when predicting whether someone would attempt suicide within the next two years, and 92 per cent accuracy in predicting whether someone would attempt suicide within the next week. When assessing the likelihood of suicide of 12,695 randomly selected hospitalised patients with no documented history of suicide attempts, the group was able to achieve even higher rates of prediction. With such results, we shouldn’t be surprised that Facebook has introduced its own proprietary AI system to detect those at increased risk of suicide.

The shortcomings of ToM have long been part of public lore – particularly in the criticism of psychiatry. But we have persisted with the belief that the fault is with psychiatry and psychiatrists, not with the basic tenet that we can know what another is thinking and feeling. For me, the final straw, the most unequivocal indictment of ToM, has been the evidence from recent political developments – from the inability to get a handle on Kim Jong-un’s mindset and nuclear intentions, to the near-universal failure of political pundits to recognise the pent-up anger, fear and resentment simmering in future Trump supporters.

I’ve got to confess, doubts about ToM began early in my neurology career. A young Jamaican woman had strangled her 18-month-old daughter to death. When sent to the psych ward of San Francisco General Hospital for observation, she had attacked a moaning, wheelchair-bound woman with dementia, breaking her neck before nearby guards could intervene (the victim died of her injuries). The court-appointed psychiatrist wanted to know if this woman’s episodes of violent behaviour might have a neurological basis.

The woman wasn’t at all like the person I’d pictured from reading her chart. With her bright smile, easy laugh and lilting accent, she was utterly engaging. I couldn’t imagine her hurting anyone, let alone her child. As expected, the hour-long exam revealed no clues to explain her behaviour. Before leaving, I gathered up my nerve and asked her if she had any idea why she’d choked her daughter and attacked the old woman.

For a long time, the woman sat motionless. She finally blurted out: ‘I hate the sound of crying.’ She folded her hands in her lap and stared at me, shaking her head. Each of us was speechless, acutely aware of the unbridgeable gap between us. I was shaken by the realisation that whatever I thought crying triggered in this woman, it would be pure fiction, a story I’d dream up to give some sense of explanation to the inexplicable.

This wasn’t an isolated event. During my career, I’ve been baffled enough times to fully accept how little access I have to the inner workings of other minds. When a patient died of a mysterious illness, I asked his 30something son permission for an autopsy. The man agreed on the condition that he was allowed to watch. When I asked him why, his sole explanation was: ‘He’s my father.’

A middle-aged woman collapsed during the middle of the night. The CAT scan revealed a massive brain haemorrhage that would almost certainly be fatal within hours. When I told her husband, he blinked a couple times, then said without any apparent emotion: ‘OK. I think I’ll go home and take a shower.’

But the most singularly illustrative demonstration of the limits of ToM occurred during the psychiatry segment of my neurology board-certification oral exam. My test patient was an unkempt man who smelled of mildew.

‘How long have you been in the hospital?’ I began the interview.

‘Three months.’

Surprised that he hadn’t been tidied up, I asked again.

‘A couple years, give or take. Time is elusive when nothing’s happening.’

‘Could you be more specific?’

‘If you pressed me, I’d say most likely I’ve been here three days.’

‘Do you have any prior history of mental illness?’

‘Who doesn’t?’

‘Other members of the family as well?’

‘Depends on who you ask.’

‘Do you have any idea why you’re here?’

‘No. Do you?’

‘Yes. You’re my test patient for the psychiatry portion of the neurology boards. It’d sure be helpful if you at least tried to give me some straight answers.’

‘Personal answers are never straight. You learn to couch yourself in yes, but, maybe, and on the other hand. You never know when you might be asked to run for president.’

Hamlet and Anna Karenina are artistic one-offs based not on deep understanding, but on the yarns we spin

And so it went – 30 minutes of agonising head shakes and bobbing and weaving, while my evaluating psychiatrist took copious notes, then indicated that time was up. He excused the patient.

‘So,’ the examiner asked. ‘What do you think?’

‘I have no idea. The patient is entirely unreliable.’

‘Surely you have some hunches.’

‘Not really. I can’t even tell if he’s putting me on.’

‘If passing or failing depends on making a diagnosis, what would you say?’

‘Sorry. I’d just be guessing.’

‘You’re dismissed,’ the psychiatrist said with a blank expression that I couldn’t interpret.

That evening, after the exam was over, I ran into the psychiatrist. He was all smiles. ‘Nice going – you passed with flying colours.’

‘You’re kidding? I was sure my psych exam was a complete whiff.’

The examiner laughed.

‘So, what was wrong with him?’ I asked.

‘Who knows? He’s one of our best; we use him for many of the exams in this part of the country.’

‘He’s a professional patient?’

‘Not exactly. He was formerly hospitalised – though no one was quite sure what was wrong. While on the ward, he acquired this uncanny ability to mimic most major psychiatric diseases. This time, we asked him to portray an uncooperative and unreliable patient.’

‘So he does have an underlying mental disorder?’

The examiner simultaneously shrugged and smiled. ‘Have a safe flight home.’

I’ve concluded that tragedy can create otherwise unimaginable responses. This is hardly mind-reading. Conjuring a different view of the world is a rare talent requiring an extraordinary leap of imagination: Hamlet, Madame Bovary and Anna Karenina are artistic one-offs based not on deep understanding, but yarns we spin about each other’s intentions and motivations. We make up stories about our spouses, our kids, our leaders, and our enemies. Inspiring narratives get us through dark nights and tough times, but we’ll always make better predictions guided by the impersonal analysis of big data than by the erroneous belief that we can read another’s mind.