Dr David Pollens is a psychoanalyst who sees his patients in a modest ground-floor office on the Upper East Side of Manhattan, a neighbourhood probably only rivalled by the Upper West Side for the highest concentration of therapists anywhere on the planet. Pollens, who is in his early 60s, with thinning silver hair, sits in a wooden armchair at the head of a couch; his patients lie on the couch, facing away from him, the better to explore their most embarrassing fears or fantasies. Many of them come several times a week, sometimes for years, in keeping with analytic tradition. He has an impressive track record treating anxiety, depression and other disorders in adults and children, through the medium of uncensored and largely unstructured talk.

To visit Pollens, as I did one dark winter’s afternoon in 2015, is to plunge immediately into the arcane Freudian language of “resistance” and “neurosis”, “transference” and “counter-transference”. He exudes a sort of warm neutrality; you could easily imagine telling him your most troubling secrets. Like other members of his tribe, Pollens sees himself as an excavator of the catacombs of the unconscious: of the sexual drives that lurk beneath awareness; the hatred we feel for those we claim to love; and the other distasteful truths about ourselves we don’t know, and often don’t wish to know.

But there’s a very well-known narrative when it comes to therapy and the relief of suffering – and it leaves Pollens and his fellow psychoanalysts decisively on the wrong side of history. For a start, Freud (this story goes) has been debunked. Young boys don’t lust after their mothers, or fear their fathers will castrate them; adolescent girls don’t envy their brothers’ penises. No brain scan has ever located the ego, super-ego or id. The practice of charging clients steep fees to ponder their childhoods for years – while characterising any objections to this process as “resistance”, demanding further psychoanalysis – looks to many like a scam. “Arguably no other notable figure in history was so fantastically wrong about nearly every important thing he had to say” than Sigmund Freud, the philosopher Todd Dufresne declared a few years back, summing up the consensus and echoing the Nobel prize-winning scientist Peter Medawar, who in 1975 called psychoanalysis “the most stupendous intellectual confidence trick of the 20th century”. It was, Medawar went on, “a terminal product as well – something akin to a dinosaur or a zeppelin in the history of ideas, a vast structure of radically unsound design and with no posterity.”

A jumble of therapies emerged in Freud’s wake, as therapists struggled to put their endeavours on a sounder empirical footing. But from all these approaches – including humanistic therapy, interpersonal therapy, transpersonal therapy, transactional analysis and so on – it’s generally agreed that one emerged triumphant. Cognitive behavioural therapy, or CBT, is a down-to-earth technique focused not on the past but the present; not on mysterious inner drives, but on adjusting the unhelpful thought patterns that cause negative emotions. In contrast to the meandering conversations of psychoanalysis, a typical CBT exercise might involve filling out a flowchart to identify the self-critical “automatic thoughts” that occur whenever you face a setback, like being criticised at work, or rejected after a date.

CBT has always had its critics, primarily on the left, because its cheapness – and its focus on getting people quickly back to productive work – makes it suspiciously attractive to cost-cutting politicians. But even those opposed to it on ideological grounds have rarely questioned that CBT does the job. Since it first emerged in the 1960s and 1970s, so many studies have stacked up in its favour that, these days, the clinical jargon “empirically supported therapies” is usually just a synonym for CBT: it’s the one that’s based on facts. Seek a therapy referral on the NHS today, and you’re much more likely to end up, not in anything resembling psychoanalysis, but in a short series of highly structured meetings with a CBT practitioner, or perhaps learning methods to interrupt your “catastrophising” thinking via a PowerPoint presentation, or online.

Yet rumblings of dissent from the vanquished psychoanalytic old guard have never quite gone away. At their core is a fundamental disagreement about human nature – about why we suffer, and how, if ever, we can hope to find peace of mind. CBT embodies a very specific view of painful emotions: that they’re primarily something to be eliminated, or failing that, made tolerable. A condition such as depression, then, is a bit like a cancerous tumour: sure, it might be useful to figure out where it came from – but it’s far more important to get rid of it. CBT doesn’t exactly claim that happiness is easy, but it does imply that it’s relatively simple: your distress is caused by your irrational beliefs, and it’s within your power to seize hold of those beliefs and change them.

Psychoanalysts contend that things are much more complicated. For one thing, psychological pain needs first not to be eliminated, but understood. From this perspective, depression is less like a tumour and more like a stabbing pain in your abdomen: it’s telling you something, and you need to find out what. (No responsible GP would just pump you with painkillers and send you home.) And happiness – if such a thing is even achievable – is a much murkier matter. We don’t really know our own minds, and we often have powerful motives for keeping things that way. We see life through the lens of our earliest relationships, though we usually don’t realise it; we want contradictory things; and change is slow and hard. Our conscious minds are tiny iceberg-tips on the dark ocean of the unconscious – and you can’t truly explore that ocean by means of CBT’s simple, standardised, science-tested steps.

This viewpoint has much romantic appeal. But the analysts’ arguments fell on deaf ears so long as experiment after experiment seemed to confirm the superiority of CBT – which helps explain the shocked response to a study, published in May 2015, that seemed to show CBT getting less and less effective, as a treatment for depression, over time.

Examining scores of earlier experimental trials, two researchers from Norway concluded that its effect size – a technical measure of its usefulness – had fallen by half since 1977. (In the unlikely event that this trend were to persist, it could be entirely useless in a few decades.) Had CBT somehow benefited from a kind of placebo effect all along, effective only so long as people believed it was a miracle cure?

That puzzle was still being digested when researchers at London’s Tavistock clinic published results in October 2015 from the first rigorous NHS study of long-term psychoanalysis as a treatment for chronic depression. For the most severely depressed, it concluded, 18 months of analysis worked far better – and with much longer-lasting effects – than “treatment as usual” on the NHS, which included some CBT. Two years after the various treatments ended, 44% of analysis patients no longer met the criteria for major depression, compared to one-tenth of the others. Around the same time, the Swedish press reported a finding from government auditors there: that a multimillion pound scheme to reorient mental healthcare towards CBT had proved completely ineffective in meeting its goals.

Such findings, it turns out, aren’t isolated – and in their midst, a newly emboldened band of psychoanalytic therapists are pressing the case that CBT’s pre-eminence has been largely built on sand. Indeed, they argue that teaching people to “think themselves to wellness” might sometimes make things worse. “Every thoughtful person knows that self-understanding isn’t something you get from the drive-thru,” said Jonathan Shedler, a psychologist at the University of Colorado medical school, who is one of CBT’s most unsparing critics. His default bearing is one of wry good humour, but exasperation ruffled his demeanour whenever our conversation dwelt too long on CBT’s claims of supremacy. “Novelists and poets seemed to have understood this truth for thousands of years. It’s only in the last few decades that people have said, ‘Oh, no, in 16 sessions we can change lifelong patterns!’” If Shedler and others are right, it may be time for psychologists and therapists to re-evaluate much of what they thought they knew about therapy: about what works, what doesn’t, and whether CBT has really consigned the cliche of the chin-stroking shrink – and with it, Freud’s picture of the human mind – to history. The impact of such a re-evaluation could be profound; eventually, it might even change how millions of people around the world are treated for psychological problems.

How does that make you feel?

* * *

“Freud was full of horseshit!” the therapist Albert Ellis, arguably the progenitor of CBT, liked to say. It’s hard to deny he had a point. One big part of the problem for psychoanalysis has been the evidence that its founder was something of a charlatan, prone to distorting his findings, or worse. (In one especially eye-popping case, which only came to light in the 1990s, Freud told a patient, the American psychiatrist Horace Frink, that his misery stemmed from an inability to recognise that he was homosexual – and hinted that the solution lay in making a large financial contribution to Freud’s work.)

But for those challenging psychoanalysis with alternative approaches to therapy, even more troublesome was the sense that even the most sincere psychoanalyst is always engaged in a guessing-game, always prone to finding “proof” of his or her hunches, whether it’s there or not. The basic premise of psychoanalysis, after all, is that our lives are ruled by unconscious forces, which speak to us only indirectly: through symbols in dreams, “accidental” slips of the tongue, or through what infuriates us about others, which is a clue to what we can’t face in ourselves. But all this makes the whole thing unfalsifiable. Protest to your shrink that, no, you don’t really hate your father, and that just shows how desperate you must be to avoid admitting to yourself that you do.

This problem of self-fulfilling prophecies is a disaster for anyone hoping to figure out, in a scientific way, what’s really going on in the mind – and by the 1960s, advances in scientific psychology had reached a point at which patience with psychoanalysis began to run out. Behaviourists such as BF Skinner had already shown that human behaviour could be predictably manipulated, much like that of pigeons or rats, by means of punishment and reward. The burgeoning “cognitive revolution” in psychology held that goings-on inside the mind could be measured and manipulated too. And since the 1940s, there had been a pressing need to do so: thousands of soldiers returning from the second world war exhibited emotional disturbances that cried out for rapid, cost-effective treatment, not years of conversation on the couch.

Before laying the groundwork for CBT, Albert Ellis had in fact originally trained as a psychoanalyst. But after practising for some years in New York in the 1940s, he found his patients weren’t getting better – and so, with a self-confidence that would come to define his career, he concluded that analysis, rather than his own abilities, must be to blame. Along with other like-minded therapists, he turned instead to the ancient philosophy of Stoicism, teaching clients that it was their beliefs about the world, not events themselves, that distressed them. Getting passed over for a promotion might induce unhappiness, but depression came from the irrational tendency to generalise from that single setback to an image of oneself as an all-round failure. “As I see it,” Ellis told an interviewer decades later, “psychoanalysis gives clients a cop-out. They don’t have to change their ways … they get to talk about themselves for 10 years, blaming their parents and waiting for magic-bullet insights.”

Thanks to the breezy, no-nonsense tone adopted by CBT’s proponents, it’s easy to miss how revolutionary its claims were. For traditional psychoanalysts – and those who practise newer “psychodynamic” techniques, largely derived from traditional psychoanalysis – what happens in therapy is that seemingly irrational symptoms, such as the endless repetition of self-defeating patterns in love or work, are revealed to be at least somewhat rational. They’re responses that made sense in the context of the patient’s earliest experience. (If a parent abandoned you, years ago, it’s not so strange to live in constant dread that your spouse might do so too – and thus to act in ways that screw up your marriage as a result.) CBT flips that on its head. Emotions that might appear rational – such as feeling depressed about what a catastrophe your life is – stand exposed as the result of irrational thinking. Sure, you lost your job; but it doesn’t follow that everything will be awful forever.

Illustration: Peter Gamlen

If this second approach is right, change is clearly far simpler: you need only identify and correct various thought-glitches, rather than decoding the secret reasons for your suffering. Symptoms such as sadness or anxiety aren’t necessarily meaningful clues to long-buried fears; they’re intruders to be banished. In analysis, the relationship between therapist and patient serves as a kind of petri dish, in which the patient re-enacts her habitual ways of relating with others, enabling them to be better understood. In CBT, you’re just trying to get rid of a problem.

The sweary, freewheeling Ellis was destined to remain an outsider, but the approach he pioneered soon attained respectability thanks to Aaron Beck, a sober-minded psychiatrist at the University of Pennsylvania. (Now 94, Beck has probably never called anything “horseshit” in his life.) In 1961, Beck devised a 21-point questionnaire, known as the Beck Depression Inventory, to quantify clients’ suffering – and showed that, in about half of all cases, a few months of CBT relieved the worst symptoms. Objections from analysts were dismissed, with some justification, as the complaints of people trying to protect their lucrative turf. They found themselves compared to 19th-century medical doctors – bungling improvisers, threatened and offended by the notion that their mystical art could be reduced to a sequence of evidence-based steps.

Many more studies followed, demonstrating the benefits of CBT in treating everything from depression to obsessive-compulsive disorder to post-traumatic stress. “I went to the early seminars on cognitive therapy to satisfy myself that it was another approach that wouldn’t work,” David Burns, who went on to popularise CBT in his worldwide bestseller Feeling Good, told me in 2010. “But I passed the techniques to my patients – and people who’d seemed hopeless and stuck for years began to recover.”

There’s little doubt that CBT has helped millions, at least to some degree. This has been especially true in the UK since the economist Richard Layard, a vigorous CBT evangelist, became Tony Blair’s “happiness czar”. By 2012, more than a million people had received free therapy as a result of the initiative Layard helped push through, working with the Oxford psychologist David Clark. Even if CBT wasn’t particularly effective, you might argue, that kind of reach would count for a lot. Yet it’s hard to shake the sense that something big is missing from its model of the suffering mind. After all, we experience our own inner lives, and our relationships with others, as bewilderingly complex. Arguably the entire history of both religion and literature is an attempt to grapple with what it all means; neuroscience daily reveals new subtleties in the workings of the brain. Could the answer to our woes really be something as superficial-sounding as “identifying automatic thoughts” or “modifying your self-talk” or “challenging your inner critic”? Could therapy really be so straightforward that you could receive it not from a human but from a book, or a computer?

A few years ago, after CBT had started to dominate taxpayer-funded therapy in Britain, a woman I’ll call Rachel, from Oxfordshire, sought therapy on the NHS for depression, following the birth of her first child. She was sent first to sit through a group PowerPoint presentation, promising five steps to “improve your mood”; then she received CBT from a therapist and, in between sessions, via computer. “I don’t think anything has ever made me feel as lonely and isolated as having a computer program ask me how I felt on a scale of one to five, and – after I’d clicked the sad emoticon on the screen – telling me it was ‘sorry to hear that’ in a prerecorded voice,” Rachel recalled. Completing CBT worksheets under a human therapist’s guidance wasn’t much better. “With postnatal depression,” she said, “you’ve gone from a situation in which you’ve been working, earning your own money, doing interesting things – and suddenly you’re at home on your own, mostly covered in sick, with no adult to talk to.” What she needed, she sees now, was real connection: that fundamental if hard-to-express sense of being held in the mind of another person, even if only for a short period each week.

“I may be mentally ill,” Rachel said, “but I do know that a computer does not feel bad for me.”

* * *

Jonathan Shedler remembers where he was when he first realised there might be something to the psychoanalytic idea of the mind as a realm far more complex, and peculiar, than most of us imagine. He was an undergraduate, at college in Massachusetts, when a psychology lecturer astonished him by interpreting a dream Shedler had related – about driving on bridges over lakes, and trying on hats in a shop – as an expression of the fear of pregnancy. The lecturer was exactly right: Shedler and his girlfriend, whose dream it was, were at that moment waiting to learn if she was pregnant, and desperately hoping she wasn’t. But the lecturer knew none of this context; he was apparently just an expert interpreter of the symbolism of dreams. “The impact could not have been greater,” Shedler recalled, if his “words had been heralded by celestial trumpets.” He decided that “if there were people in the world who understood such things, I had to be one of them.”

Yet academic psychology, the field Shedler next entered, meant having that kind of enthusiasm for the mysteries of the mind drummed out of you; researchers, he concluded, were committed to quantification and measurement, but not to the inner lives of real people. To become a psychoanalyst takes years of training, and it’s compulsory to undergo analysis yourself; studying the mind at university, by contrast, requires zero real-life experience. (Shedler is now that rarity, a trained therapist and researcher, who bridges both worlds.) “You know that thing about how you need 10,000 hours of practice to develop an expertise?” he asked. “Well, most of the researchers making pronouncements [about which therapies work] don’t have 10 hours!”

Shedler’s subsequent research and writing has played a significant role in undermining the received wisdom that there’s no hard evidence for psychoanalysis. But it’s undeniable that the early psychoanalysts were sniffy about research: they were prone to viewing themselves as embattled practitioners of a subversive art that needed nurturing in specialist institutions – which in practice meant forming cliquish private bodies, and rarely interacting with university experimenters. Research into cognitive approaches thus got a big head start – and it was the 1990s before empirical studies of psychoanalytic techniques began hinting that the cognitive consensus might be flawed. In 2004, a meta-analysis concluded that short-term psychoanalytic approaches were at least as good as other routes for many ailments, leaving recipients better off than 92% of all patients prior to therapy. In 2006, a study tracking approximately 1,400 people suffering from depression, anxiety and related conditions ruled in favour of short-term psychodynamic therapy, too. And a 2008 study into borderline personality disorder concluded that only 13% of psychodynamic patients still had the diagnosis five years after the end of treatment, compared with 87% of the others.

These studies haven’t always compared analytic therapies with cognitive ones; the comparison is often with “treatment as usual”, a phrase that covers a multitude of sins. But again and again, as Shedler has argued, the starkest differences between the two emerge some time after therapy has finished. Ask how people are doing as soon as their treatment ends, and CBT looks convincing. Return months or years later, though, and the benefits have often faded, while the effects of psychoanalytic therapies remain, or have even increased – suggesting that they may restructure the personality in a lasting way, rather than simply helping people manage their moods. In the NHS study conducted at the Tavistock clinic, chronically depressed patients receiving psychoanalytic therapy stood a 40% better chance of going into partial remission, during every six-month period of the research, than those receiving other treatments.

Alongside this growing body of evidence, scholars have begun to ask pointed questions about the studies that first fuelled CBT’s ascendancy. In a provocative 2004 paper, the Atlanta-based psychologist Drew Westen and his colleagues showed how researchers – motivated by the desire for an experiment with clearly interpretable results – had often excluded up to two-thirds of potential participants, typically because they had multiple psychological problems. The practice is understandable: when a patient has more than one problem, it’s harder to untangle the lines of cause and effect. But it may mean that the people who do get studied are extremely atypical. In real life, our psychological problems are intricately embedded in our personalities. The issue you bring to therapy (depression, say) may not be the one that emerges after several sessions (for example, the need to come to terms with a sexual orientation you fear your family won’t accept). Moreover, some studies have sometimes seemed to unfairly stack the deck, as when CBT has been compared with “psychodynamic therapy” delivered by graduate students who’d received only a few days’ cursory training in it, from other students.

But the most incendiary charge against cognitive approaches, from the torchbearers of psychoanalysis, is that they might actually make things worse: that finding ways to manage your depressed or anxious thoughts, for example, may simply postpone the point at which you’re driven to take the plunge into self-understanding and lasting change. CBT’s implied promise is that there’s a relatively simple, step-by-step way to gain mastery over suffering. But perhaps there’s more to be gained from acknowledging how little control – over our lives, our emotions, and other people’s actions – we really have? The promise of mastery is seductive not just for patients but therapists, too. “Clients are anxious about being in therapy, and inexperienced therapists are anxious because they don’t have a clue what to do,” writes the US psychologist Louis Cozolino in a new book, Why Therapy Works. “Therefore, it is comforting for both parties to have a task they can focus on.”

Not surprisingly, leading proponents of CBT reject most of these criticisms, arguing that it’s been caricatured as superficial, and that some decrease in effectiveness is only to be expected, because it’s grown so much in popularity. Early studies used small samples and pioneering therapists, enthused by the new approach; more recent studies use bigger samples, and inevitably involve therapists with a wider range of talent levels. “People who say CBT is superficial have just missed the point,” said Trudie Chalder, professor of cognitive behavioural psychotherapy at the King’s College Institute of Psychiatry, Psychology and Neuroscience in London, who argues that no single therapy is best for all maladies. “Yes, you’re targeting people’s beliefs, but you’re not just targeting easily accessible beliefs. It’s not just ‘Oh, that person looked at me peculiarly, so they must not like me’; it’s beliefs like ‘I’m an unlovable person’, which may derive from early experience. The past is very much taken into account.”

Nonetheless, the dispute won’t be settled by adjudicating between clashing studies: it goes deeper than that. Experimenters may reach wildly different conclusions about which therapies have the best outcomes. But what should count as a successful outcome anyway? Studies measure relief of symptoms – yet a crucial premise of psychoanalysis is that there’s more to a meaningful life than being symptom-free. In principle, you might even end a course of psychoanalysis sadder – though wiser, more conscious of your previously unconscious responses, and living in a more engaged way – and still deem the experience a success. Freud famously declared that his goal was the transformation of “neurotic misery into common unhappiness”. Carl Jung said “humanity needs difficulties: they are necessary for health.” Life is painful. Should we be thinking in terms of a “cure” for painful emotions at all?

* * *

There’s something deeply appealing about the idea that therapy shouldn’t be approached as a matter of science – that our individual lives are too distinctive to be submitted to the relentless generalisation by which science must proceed. That sentiment may help explain the commercial success of The Examined Life, Stephen Grosz’s 2013 collection of tales from the analyst’s couch, which spent weeks on UK bestseller lists and has been translated into more than 30 languages. Its chapters consist not of experimental findings or clinical diagnoses, but of stories, many of which involve a jolt of insight as the patient suddenly gets a sense of the depths he or she contains. There’s the man who lies compulsively, in a bid for secret intimacy with those he can persuade to join him in deceit, just like his mother hid evidence of his bedwetting; and the woman who finally realises how effortfully she’s been denying the evidence of her husband’s infidelity when she notices how neatly someone has stacked the dishwasher.

“Each life is unique, and your role, as an analyst, is to find the unique story of the patient,” Grosz told me. “There are so many things that only come out through slips of the tongue, through someone confiding a fantasy, or using a certain word.” The analyst’s job is to stay watchfully receptive to it all – and then, from such ingredients, “help people make meaning of their lives.”

Surprisingly, perhaps, recent support for this seemingly unscientific perspective has emerged from the most empirical corner of the study of the mind: neuroscience. Many neuroscience experiments have indicated that the brain processes information much faster than conscious awareness can keep track of it, so that countless mental operations run, in the neuroscientist David Eagleman’s phrase, “under the hood” – unseen by the conscious mind in the driving-seat. For that reason, as Louis Cozolino writes in Why Therapy Works, “by the time we become consciously aware of an experience, it has already been processed many times, activated memories, and initiated complex patterns of behaviour.”

Depending on how you interpret the evidence, it would seem we can do countless complex things – from performing mental arithmetic, to hitting a car’s brakes to avoid a collision, to making a choice of marriage partner – before becoming aware that we’ve done them. This doesn’t mesh well with a basic assumption of CBT – that, with training, we can learn to catch most of our unhelpful mental responses in the act. Rather, it seems to confirm the psychoanalytic intuition that the unconscious is huge, and largely in control; and that we live, unavoidably, through lenses created in the past, which we can only hope to modify partially, slowly and with great effort.

Perhaps the only undeniable truth to emerge from disputes among therapists is that we still don’t have much of a clue how minds work. When it comes to easing mental suffering, “it’s like we’ve got a hammer, a saw, a nail-gun and a loo brush, and this box that doesn’t always work properly, so we just keep hitting the box with each of these tools to see what works,” said Jules Evans, policy director for the Centre for the History of Emotions at Queen Mary, University of London.

This may be why many scholars have been drawn to what has become known as the “dodo-bird verdict”: the idea, supported by some studies, that the specific kind of therapy makes little difference. (The name comes from the Dodo’s pronouncement in Alice in Wonderland: “Everybody has won, and all must have prizes.”) What seems to matter much more is the presence of a compassionate, dedicated therapist, and a patient committed to change; if one therapy is better than all others for all or even most problems, it has yet to be discovered. David Pollens, in his Upper East Side consulting room, said he had some sympathy for that verdict, despite his passion for psychoanalysis. “There was a wonderful British analyst, Michael Balint, who was very involved in medical training, and he had a question he liked to pose [to doctors],” Pollens said. It was: “‘What do you think is the most powerful medication you prescribe?’ And people would try to answer that, and then eventually he’d say: ‘the relationship’.”

Yet even this conclusion – that we simply don’t know which therapies work best – might be seen as a point in favour of Freud and his successors. Psychoanalysis, after all, embodies just this awed humility about how little we can ever grasp about the workings of our minds. (The one question nobody can ever answer, writes the Jungian analyst James Hollis, is “of what are you unconscious?”) Freud the man scaled heights of arrogance. But his legacy is a reminder that we shouldn’t necessarily expect life to be all that happy, nor to assume we can ever really know what’s going on inside – indeed, that we’re often deeply emotionally invested in preserving our ignorance of unsettling truths.

“What happens in therapy,” Pollens said, “is that people come in asking for help, and then the very next thing they do is they try to stop you helping them.” His smile hinted at the element of absurdity in the situation – and in the whole therapeutic undertaking, perhaps. “How do we help a person when they’ve told you, in one way or another, ‘Don’t help me’? That’s what analytic treatment is about.”