In the 1980s, Californian politician John Vasconcellos set up a task force to promote high self-esteem as the answer to all social ills. But was his science based on a lie?

In 2014, a heartwarming letter sent to year 6 pupils at Barrowford primary school in Lancashire went viral. Handed out with their Key Stage 2 exam results, it reassured them: “These tests do not always assess all of what it is that make each of you special and unique… They do not know that your friends count on you to be there for them or that your laughter can brighten the dreariest day. They do not know that you write poetry or songs, play sports, wonder about the future, or that sometimes you take care of your little brother or sister.”

At Barrowford, people learned, teachers were discouraged from issuing punishments, defining a child as “naughty” and raising their voices. The school’s guiding philosophy, said headteacher Rachel Tomlinson, was that kids were to be treated with “unconditional positive regard”.

A little more than a year later, Barrowford found itself in the news again. Ofsted had given the school one of its lowest possible ratings, finding the quality of teaching and exam results inadequate. The school, their report said, “emphasised developing pupils’ emotional and social wellbeing more than the attainment of high standards”. Somehow, it seemed, the nurturing of self-esteem had not translated into higher achievement.

The flawed yet infectious notion that, in order to thrive, people need to be treated with unconditional positivity first gained traction in the late 80s. Since then, the self-esteem movement has helped transform the way we raise our children – prioritising their feelings of self-worth, telling them they are special and amazing, and cocooning them from everyday consequences.

One manifestation of this has been grade inflation. In 2012, the chief executive of British exams regulator Ofqual admitted the value of GCSEs and A-levels had been eroded by years of “persistent grade inflation”. In the US, between the late 60s and 2004, the proportion of first year university students claiming an A average in high school rose from 18% to 48%, despite the fact that SAT scores had actually fallen. None of this, says Keith Campbell, professor of psychology at the University of Georgia and expert on narcissism, serves our youngsters well. “Burning yourself on a stove is really useful in telling you where you stand,” he says, “but we live in a world of trophies for everyone. Fourteenth place ribbon. I am not making this stuff up. My daughter got one.”

Campbell, with his colleague Jean Twenge at San Diego State University, has argued that this kind of parenting and teaching has contributed to a measurable rise in narcissism: witness the selfie-snapping millennials. Although their findings are disputed, Twenge points to other research done in the US and beyond – “twenty-two studies or samples [that] show a generational increase in positive self-views, including narcissism, and only two [that] do not”.

To get ahead in the 1980s, you had to be ruthless, relentless. You had to believe in yourself

How did we get here? To answer that, you have to go back to 1986 and the work of an eccentric and powerful California politician, John “Vasco” Vasconcellos. That year, the Democrat Vasconcellos managed to persuade a deeply sceptical Republican state governor to fund a three-year task force to explore the value of self-esteem. Vasco was convinced that low self-esteem was the source of a huge array of social issues, including unemployment, educational failure, child abuse, domestic violence, homelessness and gang warfare. He became convinced that raising the population’s self-esteem would act as a “social vaccine”, saving the state billions.

But Vasco’s plan backfired spectacularly, with the fallout lasting to this day. I spent a year trying to find out why – and discovered that there was, at the heart of his project, a lie.

***

John Vasconcellos grew up an obedient Catholic, an altar boy, the smartest kid in his class, whose mother swore that he never misbehaved. But, being such a devout Catholic, he knew that no matter how good he was, he could only ever be a sinner. At primary school, he ran for class president. “I lost by one vote. Mine,” he later said. He didn’t vote for himself because “I’d been drilled never to use the word ‘I’, never to think or speak well of myself.”

After a spell as a lawyer, Vasco entered politics. In 1966, aged 33, he was elected to the California state assembly. But there was a problem: his professional success was at odds with how he thought of himself; he felt he didn’t deserve it. At 6ft 3in and over 200lb, he would stalk the Capitol building in Sacramento, glowering and anxious in his smart black suit, perfect white shirt and arrow-straight tie, his hair cropped with military precision. “I found my identity and my life coming utterly apart,” he later said. “I had to go and seek help.”

That help came from an unusual Catholic priest: Father Leo Rock was a psychologist who had trained under the pioneer of humanistic psychology, Carl Rogers, a man who believed that the Catholics had it absolutely wrong. At their core, he thought, humans weren’t bad; they were good. And in order to thrive, people needed to be treated with “unconditional positive regard” (Rogers coined the phrase). Vasco began studying under Rogers himself, a man he later described as “almost my second father”. Through intense group therapy workshops at the Esalen Institute in Big Sur, Vasco became a devotee of the human potential movement, based partly on the Rogerian idea that all you need to do to live well is discover your authentic inner self.

Facebook Twitter Pinterest Illustration: Franck Allais/The Guardian

Around the state capitol, Vasco’s colleagues began to notice the buttoned-up Catholic was unbuttoning. He grew his hair and wore half-open Hawaiian shirts on the floor of the senate, a gold chain nestled in his chest hair. One reporter described him as looking like “a cross between a rock star and a drug smuggler”. He became a human potential evangelist, preaching the innate goodness of humans and handing long book lists to colleagues. His self-hating Catholic self had washed away, and in its place was a great, glowing letter “I”.

Vasco knew he was in a unique position. As a politician, he could take everything he’d learned about human potential and turn it into policy that would have a real effect on thousands, perhaps millions, of lives. He decided to campaign for a state-financed task force to promote self-esteem: this would give the movement official affirmation and allow politicians to fashion legislation around it. Best of all, they could recruit the world’s finest researchers to prove, scientifically, that it worked.

In the mid-80s, the notion that feeling good about yourself was the answer to all your problems sounded to many like a silly Californian fad. But it was also a period when Thatcher and Reagan were busily redesigning western society around their project of neoliberalism. By breaking the unions, slashing protections for workers and deregulating banking and business, they wanted to turn as much of human life as possible into a competition of self versus self. To get along and get ahead in this new competitive age, you had to be ambitious, ruthless, relentless. You had to believe in yourself. What Vasco was offering was a simple hack that would make you a more winning contestant.

Vasco’s first attempt at having his task force mandated into law came to a halt in 1984, when he suffered a heart attack. His belief in positive thinking was such that, in an attempt to cure himself, he wrote to his constituents asking them to picture themselves with tiny brushes swimming through his arteries, scrubbing at the cholesterol, while singing, to the tune of Row, Row, Row Your Boat: “Now let’s swim ourselves/ up and down my streams/Touch and rub and warm and melt/the plaque that blocks my streams.” It didn’t work. As the senate voted on his proposal, Vasco was recovering from seven-way coronary bypass surgery.

After a second attempt was vetoed by the state governor, Vasco decided to enhance the name of his project, upgrading it to the Task Force to Promote Self-Esteem and Personal and Social Responsibility. He reduced the proposed budget from $750,000 a year to $735,000 over three, to be spent on academic research and the gathering of evidence in the form of public testimony. On 23 September 1986, Assembly Bill 3659 was signed into law.

The response from the California media was immediate and savage. One editorial, in the San Francisco Chronicle, called Vasco’s task force “naive and absurd”. Nothing made Vasco more angry than his ideas not being taken seriously, but he was about to become the joke of America.

***

Until Monday 9 February 1987, Vasco’s task force had been largely state news. But on that morning, the cartoonist Garry Trudeau, who had been tickled by the politician’s crusade, began an extraordinary two-week run of his popular Doonesbury strip devoted to it. By the end of that day, reporters were crowding Vasco on the floor of the assembly chamber. Rival politicians gave dismissive briefings – “You could buy the Bible for $2.50 and do better” – while the Wall Street Journal’s story bore the headline Maybe Folks Would Feel Better If They Got To Split The $735,000.

Vasco's credibility turned on a single fact: that the professors had confirmed his hunch. The only problem? They hadn’t

Vasco was livid. The media, he complained, were “terrible, cynical, sceptical and cheap”. Their problem? “Low self-esteem.”

Meanwhile, something remarkable seemed to be happening. The response from the people of California had been great. Between its announcement and the task force’s first public meeting in March 1987, the office received more than 2,000 calls and letters, and almost 400 applications to volunteer. More than 300 people came forward to speak in support of self-esteem at public hearings across the state. And even if the media’s tone wasn’t always respectful, Vasco himself was now a national figure. He appeared everywhere from Newsweek to the CBS Morning Show to the BBC. This, he sensed, could be a major opportunity.

But first he needed to find a way to wrench the media conversation upwards. And things, on that front, were going from unfortunate to ridiculous. It began with the announcement of the task force’s 25 members. On the upside, it was a diverse group, including women, men, people of colour, gay people, straight people, Republicans, Democrats, a former police officer and Vietnam veteran who’d been awarded two Purple Hearts. On the downside, it also included a white man in a turban who predicted the work of the task force would be so powerful, it would cause the sun to rise in the west. A delighted Los Angeles Herald told how, in front of the press, one member of the task force had asked others to close their eyes and imagine a “self-esteem maintenance kit” of magic hats, wands and amulets.

Vasco’s team began hearing testimony from people up and down California. They heard from an LA deputy sheriff who toured schools, attempting to reduce drug use by telling pupils, “You are special. You are a wonderful individual.” They heard from masked members of the Crips, who blamed their violent criminality on low self-esteem. One school principal recommended having elementary pupils increase their self-importance by doing evaluations on their teachers. A woman called Helice Bridges explained how she’d dedicated her life to distributing hundreds of thousands of blue ribbons that read Who I Am Makes A Difference.

With the national media given so much to snigger over, it was beginning to look as if Vasco’s mission was a bust. But there had been some good news: the University of California had agreed to recruit seven professors to research the links between low self-esteem and societal ills. They would report back in two years’ time. For Vasco, their findings would be personal. If the professors decided he was wrong, it was all over.

***

Facebook Twitter Pinterest Me, myself and I: a selfie-snapping millennial. Photograph: Francois Lenoir/Reuters

At 7.30pm on 8 September 1988, Vasco met the scientists at El Rancho Inn in Millbrae, just outside San Francisco, to hear the results. Everything hinged on Dr Neil Smelser, an emeritus professor of sociology who had coordinated the work, leading a team who reviewed all the existing research on self-esteem. And the news was good: four months later, in January, the task force issued a newsletter: “In the words of Smelser, ‘The correlational findings are very positive and compelling.’”

The headlines quickly piled up: Self-Esteem Panel Finally Being Taken Seriously; Commission On Self-Esteem Finally Getting Some Respect. The state governor sent the professors’ research to his fellow governors, saying, “I’m convinced that these studies lay the foundation for a new day in American problem solving.”

Vasco’s task force was almost done: all they had to do now was build upon this positive tone with the publication of their final report, Toward A State Of Esteem, in January 1990. That report turned out to be a victory beyond the reasonable hopes of anyone who had witnessed its humiliating origins. The governor of Arkansas, Bill Clinton, who’d privately mocked Vasco and his project, now publicly endorsed it, as did figures including Barbara Bush and Colin Powell. Time magazine ran with the headline, “The sneers are turning to cheers.”

The man they were calling the Johnny Appleseed of Self-Esteem appeared on the Today Show and Nightline, on the BBC and Australia’s ABC. The report went into reprint in its debut week and went on to sell an extraordinary 60,000 copies. Vasco’s publicists approached Oprah Winfrey, who ran a prime-time special examining why she believed self-esteem was going to be one of the “catch-all phrases for the 1990s”. Interviewed were Maya Angelou, Drew Barrymore and John Vasconcellos.

What had really happened at that meeting? I found the answer on an old audio cassette, hissy and faint

Four months after the launch of Toward A State Of Esteem, the papers were reporting that self-esteem was “sweeping through California’s public schools”, with 86% of the state’s elementary school districts and 83% of high school districts implementing self-esteem programmes. In Sacramento, students began meeting twice a week to decide how to discipline other students; in Simi Valley, kids were taught, “It doesn’t matter what you do, but who you are.” Political leaders from Arkansas to Hawaii to Mississippi began considering their own task forces.

As the months became years, the self-love movement spread. Defendants in drug trials were rewarded with special key chains for appearing in court, while those who completed treatment were given applause and doughnuts. Children were awarded sports trophies just for turning up; a Massachusetts school district ordered children in gym classes to skip without actual ropes lest they suffer the self-esteem catastrophe of tripping. Meanwhile, police in Michigan seeking a serial rapist instructed the public to look out for a thirtysomething male with medium build and “low self-esteem”.

The credibility of Vasco’s task force turned largely on a single fact: that, in 1988, the esteemed professors of the University of California had analysed the data and confirmed his hunch. The only problem was, they hadn’t. When I tracked down one renegade task force member, he described what happened as “a fucking lie”. And Vasco was behind it.

***

In an attempt to discover how America, and then the world, got conned so spectacularly, I travelled to Del Mar, California, to meet the task force member who’d predicted their work would cause the sun to rise in the west. David Shannahoff-Khalsa welcomed me into his bungalow, looking little changed from the old photographs I’d seen: face narrow, eyes sharp, turban blue. A kundalini yoga practitioner who believed meditation to be an “ancient technology of the mind”, Shannahoff-Khalsa had been so disillusioned by the final report, he’d refused to sign it.

Facebook Twitter Pinterest Illustration: Franck Allais for the Guardian

As we sat and nibbled cheese, he picked up a thick book with a shiny red cover: The Social Importance Of Self-Esteem. This was the collected work of the University of California professors. He flicked through its pages, settling eventually on Smelser’s summary of the findings. “The news most consistently reported,” he read out loud, “is that the association between self-esteem and its expected consequences are mixed, insignificant or absent.”

This was a radically different conclusion from that fed to the public. Shannahoff-Khalsa told me he was present when Vasco first saw preliminary drafts of the professors’ work. “I remember him going through them – and he looks up and says, ‘You know, if the legislature finds out what’s in these reports, they could cut the funding to the task force.’ And then all of that stuff started to get brushed under the table.”

How did they do that?

“They tried to hide it. They published a [positive] report before this one,” he said, tapping the red book, which deliberately “ignored and covered up” the science.

It was hard to believe that Vasco’s task force had been so rash as simply to invent the quote, the one that stated the findings were “positive and compelling”. What had really happened at that meeting in September 1988? I found the answer on an old audio cassette in the California state archives.

The sound was hissy and faint. What I heard, though, was clear enough. It was a recording of Smelser’s presentation to Vasco’s task force at that meeting in El Rancho Inn, and it was nowhere near as upbeat as the task force had claimed. I listened as he announced the professors’ work to be complete but worryingly mixed. He talked through a few areas, such as academic achievement, and said: “These correlational findings are really pretty positive, pretty compelling.” This, then, was the quote the task force used. They’d sexed it up a little for the public. But they had completely omitted what he said next: “In other areas, the correlations don’t seem to be so great, and we’re not quite sure why. And we’re not sure, when we have correlations, what the causes might be.”

Smelser then gave the task force a warning. The data was not going to give them something they could “hand on a platter to the legislature and say, ‘This is what you’ve got to do and you’re going to expect the following kind of results.’ That is another sin,” he said. “It’s the sin of overselling. And nobody can want to do that.”

I wondered whether Smelser was angry about the quote that got used. So I called him. He told me the university got involved in the first place only because Vasco was in charge of its budget. “The pressure [from Vasco] was indirect. He didn’t say, ‘I’m going to cut your budget if you don’t do it.’ But, ‘Wouldn’t it be a good idea if the university could devote some of its resources to this problem?’” It turned out that Smelser wasn’t at all surprised about their dubious treatment of the data. “The task force would welcome all kinds of good news and either ignore or deny bad news,” he said. “I found this was a quasi-religious movement, and that’s the sort of thing that happens in those dynamics.”

Vasco passed away, aged 82, in 2014, but I traced his right-hand man, task force chairman and veteran politician Andrew Mecca. When we finally spoke, he confirmed that it was the prestige of the University of California that had turned things around for Vasco. “That earned us some credibility stripes,” he said. Like Smelser, he felt that the university became involved only out of fear of Vasco. “John chaired their lifeblood. Their budget!” he chuckled.

The appeal of narcissists: why do we love people who'd rather love themselves? Read more

How did he rate the academics’ research? “As you read the book,” he said, “it’s a bunch of scholarly gobbledegook.”

What was Mecca’s response when the data didn’t say what he wanted?

“I didn’t care,” he said. “I thought it was beyond science. It was a leap of faith. And I think only a blind idiot wouldn’t believe that self-esteem isn’t central to one’s character and health and vitality.”

Was Vasconcellos angry when he read the professors’ reports?

“The thing is, John was an incredible politician. He was pragmatic enough that he felt he had what he needed, and that was a scholarly report that pretty much said, ‘Self-esteem’s important.’ At least, that’s the spin we got in the media.”

Mecca told me that, prior to the final report’s publication, he and Vasco visited editors and television producers up and down the country, in a deliberate attempt to construct the story before it could be subverted. An extraordinary $30,000 was spent on their PR campaign: at its height, five publicists were working full time. “We decided to make sure we got out there to tell our story and not let them interpret it from the stuff that was being written by Smelser. We cultivated the message. And that positiveness prevailed.”

So nobody listened to what Smelser and Shannahoff-Khalsa were saying?

“I’m not sure anybody cared,” Mecca said. “Who remembers Neil Smelser or Shannahoff-Khalsa? Nobody! They were tiny ripples in a big tsunami of positive change.”

***

More than 20 years on, the effects of Vasco’s mission linger. Whether the tsunami of change he brought about was wholly positive remains doubtful. I spoke to educational psychologist Dr Laura Warren, who taught in British schools in the 90s, and remembers her school’s edict that staff use mauve pens to mark errors, in place of the negative red. “It was a policy of ‘reward everything that they do’,” she told me. “That turned out to be a terribly bad idea.”

The Ofsted inspectors discovered as much when they visited Barrowford primary school in 2015. But after their critical report became public, the headteacher, Rachel Tomlinson, defended herself in her local newspaper. “When we introduced the policy, it was after an awful lot of research and deliberation,” she said. “And I think it has been a success.”