Illustration by John Holcroft

The Beatles song “Getting Better” is a small masterpiece of ambiguity. It shifts between the temperaments of Paul McCartney – sunny, positive – and John Lennon – sceptical, negative. Here’s the chorus:

I’ve got to admit it’s getting better (Better)

A little better all the time (It can’t get no worse).

That final Lennonian comment undercuts the optimism and implicitly asks the killer question: better than what? By the end of the song the meliorating McCartneyisms are descending into desperate repetitions. Now ask yourself, who would you rather have dinner with – chirpy Paul or sardonic John? If you answered Paul, you’re not going to agree with any of what follows.

Here are another couple of songs, both by Noël Coward. “There Are Bad Times Just Around the Corner” was written in 1952 (Robbie Williams covered it in 1999). It gleefully spans the globe, finding bad news everywhere and lampooning the upbeat invocations of the war years. It concludes: “We’re going to unpack our troubles from our old kitbag/And wait until we drop down dead.” Then there’s “Why Must the Show Go On?”, which dates from 1956. The title asks another killer question and the whole song brutally subverts the silly sentimentalities of show business: “And if you lose hope,/Take dope,/And lock yourself in the john,/Why must the show go on?”

These three cases make one point – pessi­mism is bracing and often very funny. It is also consoling, in that it relieves us of the burdens borne by the optimist: the need to insist it’s getting better, the enervating search for good news, the rule-driven need to get the pointless job done (and, in the long run, every job is pointless).

Sadly – but to pessimists, predictably – pessimism gets a bad press. This is because it is routinely assumed to be the same as, or an inevitable aspect of, depression. As the happiest and most well-adjusted person I know is a devout pessimist, I find this idea ridiculous. My friend delights in life precisely because he expects nothing of it. If he happens upon something good or beautiful, then it is a bonus, a miracle. His days are full of discoveries and consolations. His sense of humour is Cowardian and Lennonish, a knowing nod of recognition to bad news and false hopes. One of his favourite expressions is the typically vintage “Mustn’t grumble”. He is, I need hardly add, a joy to be with.

The old-fashioned quality of that “must­n’t grumble” is important, as is the fact that those two Coward songs come from the 1950s. In the immediate postwar years the default British mood seemed to be a resilient, good-humoured pessimism shot through with something darker.

“I think,” said Kingsley Amis, a definitively postwar figure, “I’d rather have instinctive pessimism than its opposite.”

Films such as Night and the City (1950), The Third Man (1949), Black Narcissus (1947), the first two Quatermass films (1955 and 1957), the Hammer horrors and the genuinely shocking Peeping Tom (1960) suggested an appetite for the forces of destruction and unreason that flow like an underground river beneath the quotidian. The war had perhaps provided further justification for our darkly ironic native sensibility.

That’s all gone now, because we are no longer allowed to be dark, ironic or, indeed, pessimistic. Neo-optimism is now as brutally enforced in Britain as it is in the United States. “In America, optimism has become almost like a cult,” the social psychologist Aaron Sackett told Psychology Today. “In this country,” says another American psychologist, “pessimism comes with a deep stigma.”

As with any cult, even reluctant individuals are forced to conform. In the same Psychology Today article, B Cade Massey, a professor of organisational behaviour at Yale, says: “It has gotten to the point where people feel pressure to think and talk in an optimistic way.” Massey’s research shows that, when assessing the risks of investments or surgical procedures, people make predictions they know are overly optimistic just because they want to belong, even in life- or wealth-threatening crises, to the clan of idiot grinning optimists who seem to be in charge.

You can feel this pressure wherever you go, notably on the internet, the multiplicity of which is anchored by a single, ferociously imposed neo-optimistic orthodoxy. The “Like” button on Facebook is one weapon of the neos. As a University of Leicester study found, it “directs debate on the social media platform in the direction of the blandly positive”. Social media, with their chattering pursuit of “likes”, followers, comments and shares, are overwhelmingly biased in the direction of an airheaded, cringe-inducing positivity. Look at the breathless Twitter feeds that babble about the sheer wonderfulness of everything, or the groups on Facebook and elsewhere consisting of people gathering together to save the world and spread niceness by, er, gathering together. Daily I get email demands to “help X celebrate” their birthday/promotion/whatever. “Help celebrate” – really?

This is not just irritating, it’s sinister. Clickbait websites are infecting once serious news media simply because they have the ability to rack up clicks in the millions by making people feel optimistic with or amused by pictures of cats. The new books editor of BuzzFeed, meanwhile, once said he would be publishing nothing but positive reviews, rejecting the “scathing takedown rip” he saw in “old media-type places”. To give this approach its proper name – this is ideology-driven censorship.

What is truly sinister here is that the internet, thanks to its useful idiot users, is being turned into a gigantic corporate shill. Be optimistic and then buy stuff and look at our adverts. Or, to put it another way, become increasingly exhausted, ignorant, poor and depressed.

Traditional media are just as bad, if less covert. There are talent and “reality” shows in which it is necessary to be overwhelmed by the joy, the glory, of taking part, even in defeat; a TV trope imported from the US where to be anything other than infuriatingly upbeat is to be deeply suspect.

John Updike described the US as “a vast conspiracy to make you happy”. What he did not add was that this conspiracy involved a fierce judgementalism and a puritan need to shun the pessimist and the grouch. (It was once suggested to me, by a Catholic, that the hysterical fever of happiness on display on US television was a legacy of Calvinism, in that the overly happy ones were advertising their membership in the predestined ranks of the saved. Maybe.)

In the UK, our fabulously amoral, clever and knowing advertising industry now avoids the honest suggestion that you should buy a product for its qualities, preferring instead a succession of cosy gags, little domestic heavens or, following those John Lewis Christmas ads, whole life stories evoking an optimistic future in which the products are essential players.

But the most insidious and effective import of neo-optimism came in the form of management “theory”. The standard text on this is Barbara Ehrenreich’s caustic, funny, readable and withering Smile or Die: How Positive Thinking Fooled America and the World. She points out that the neo-optimism to which we are now subjected is not, as many claim, some foundational American value. The US Declaration of Independence and the US constitution are neither pessimistic nor optimistic: they are realistic – above all, about human nature. Furthermore, as Max Weber understood, there is nothing intrinsically optimistic about capitalism; it is grinding, risk-laden, hard work, worthwhile because, to the Protestant imagination, it is God’s work.

This began to change in the 19th century with the rise of what came to be known as positive thinking; a rejection, Ehrenreich says, of the austerities of Calvinism. This did have distinguished origins in the work of Ralph Waldo Emerson and William James. In our time, however, it has descended into cultish hucksterism.

Modern-day positive thinking, like British “mustn’t grumble” pessimism, came into its own in the 1950s with the publication of Norman Vincent Peale’s book The Power of Positive Thinking (1952). Thus it pre-dated – and certainly inspired – the snake-oil wave of management theory that really got going in the 1960s. This view of the world makes perpetual economic growth and infinite life enhancement seem not only possible, but also ordained. The cruel flip side of this is that failure will be seen as a refusal to think positively and, therefore, the poor and the excluded are not unfortunate or persecuted: they are guilty.

“If optimism is the key to material success,” Ehrenreich says, “and if you can achieve an optimistic outlook through the discipline of positive thinking, then there is no excuse for failure.”

Political quietism combined with a futile and febrile pursuit of material fulfilment is the result. As she then points out, this attitude must ultimately be founded on the preposterous idea that your state of mind can change the world and overcome the contingencies of life, if not (yet) death. This is superstition, as is the entire positive thinking industry, the scale of which is chilling. Americans apparently spend more than $100bn a year on motivating their employees using various positive thinking techniques.

This is madness, as anybody who has been subjected to team-building or any of the other devices from the shabby book of spells that is management theory will attest. It produces palpably false statements such as this one from Marc Andreessen, a Silicon Valley entrepreneur and investor: “And I can tell you, at least from the last 20 years, if you bet on the side of the optimists, generally you’re right.” In fact, once you take into account the number of optimistic failures, you’d lose every penny.

More preposterously, there was the supreme expression of positive thinking that was The Secret (2006), a book by Rhonda Byrne. This exposed the superstitious roots of positive thinking by openly saying that there was a “law of attraction”, whereby the universe would materially reward your positive thoughts. Our own dear Noel Edmonds is an adherent of something similar called “cosmic ordering”, a form of intergalactic Amazon.

That this has got dangerously out of hand is obvious to the most intelligent. The Nobel Prizewinner Daniel Kahneman (the author of the bestseller Thinking, Fast and Slow) and his collaborator Dan Lovallo point out that optimism undermines executive decisions. They show that forecasts based solely on internal company attitudes are often wildly overoptimistic and suggest that companies should instead adopt “reference class forecasting”, where the performance of outsiders in similar situations is taken into account, and at once pessimism intrudes. There is also the Icarus paradox, identified by the economist Danny Miller, which is all about the way extreme success in business is often followed by abject failure, precisely because of the overoptimism fomented by the good times.

In politics neo-optimism can be lethal. As John O’Sullivan has pointed out, Blair and Brown were inveterate optimists – in foreign policy and finance – and their legacy can be seen in the chaos of the Iraq war and its aftermath, and in the crime wave that engulfed the City of London. The giant Ponzi scheme that was the financial system up to 2007 (and maybe still is) was based on optimism that was either cynical, in the case of the banks, or naive, in the case of their victims. Blair’s optimism was founded on the bizarre neoconservative conviction that we in particular could, by force of arms, bully the world into liberal democracy.

Yet dumb optimism is now the default mode in politics. Who, now, could say like Churchill, “I have nothing to offer but blood, toil, tears and sweat”? And note the word “nothing”. This was not a tem­porary state of affairs: there was no optimism available.

There are two areas in which neo-optimism seems to be more firmly grounded – medicine and history. There have been many medical studies in which the attitude of the patient appears to affect the course of an illness. In some cases this has inspired yet more superstition; when mood was found to have a marginal effect on the immune system there was a rash of claims that optimism could cure cancer.

The jury is out on this but my own suspicion is that the guilty party here may be not pessimism, but depression. Besides, happy, calm, deluded and optimistic patients sound suspiciously convenient for the medical profession. Again, large sums of money are involved and scepticism is not, therefore, permissible.

Cracks are emerging, however, in the façade of medical optimism. People are noticing that the ways of measuring such traits – pencil-and-paper tests – are dubious and that the assumption that these are inborn traits that follow you through life may be wrong. People may be strategically optimistic or pessimistic according to the situation in which they find themselves. This would imply that pessimism has adaptive advantages: a prime heresy but, on reflection, very likely indeed.

Neo-optimistic history, meanwhile, is really an aspect of scientism, the belief that to every coherent question, there must be a scientific answer. I noticed this a few years ago at a Penguin dinner attended by, among others, Steven Pinker, David Deutsch and Simon Baron-Cohen. The mood, particularly in the case of Pinker and Deutsch, was optimism, supported by the belief that, thanks to the Enlightenment, we have (in theory at least) cracked most of our outstanding problems. The book Pinker was there to plug, The Better Angels of Our Nature: a History of Violence and Humanity, has since become one of the two bibles (the other being Richard Dawkins’s The God Delusion) of the scientistic faith and of science-based neo-optimism.

Pinker’s case is that violence, according to crime rates and war deaths, is in long-term decline. He attributes this, at least in part, to the spread of Enlightenment rationality. But technically he is not being optimistic, because he does not aspire to make forecasts. There is no evidence that this trend will continue. Nevertheless, there seems to be something McCartneyish at work here. My own first reaction was that nuclear weapons have certainly prevented a huge number of battlefield deaths but that just means we have concentrated our violence in terrible weapons that might be used at any moment. Future violence could, in an instant, turn out to be far worse than anything in the past.

There are now also many non-battlefield deaths – in Congo, for instance, or increasingly under the rule of Islamic State – that are caused by war. Furthermore, a recent paper by the political scientist Tanisha Fazal calls many of Pinker’s statistics into question. War deaths, she shows, have been prevented by better medical facilities, quicker extraction of soldiers from the battlefield and the better health of combat troops to start with. This evidence does not refute Pinker but it does make some of his figures seem less startling.

Scientific or quasi-scientific neo-optimism is the default mode of our time. Things will get better, it is believed and, looking back on the dark, pre-Enlightenment centuries, Lennon was right: things can’t get much worse. This is obviously historically illiterate. The 20th century happened most disastrously to Germany, previously the best-educated, most enlightened country in Europe. Furthermore, optimism was not actually an Enlightenment value. Voltaire, that prince of the cause, despised optimism because of the brutal contingencies of his own life and because of the even more brutal contingencies of nature – the Lisbon earthquake of 1755 in particular, in which up to 100,000 people died. Turning your newly rational gaze on the realities of the world should deliver a healthy shot of pessimism. Or, as Saul Bellow put it, casually refuting Socrates’s best-known motto, “the overexamined life might make you wish you were dead”.

All of which matters: first, because unalloyed optimism is dangerous, in personal life as much as in politics and business. The best advice ever given was to be positive yet to expect the worst. “Put your trust in God, my boys, and keep your powder dry,” was how it was said in Cromwell’s era (not, as legend suggests, by Cromwell). “Trust, but verify,” was Ronald Reagan’s version; he was quoting a Russian proverb.

Second, it matters because of the idiot optimism that has become the default mode of much modern culture – from breakfast TV to game, talent and reality shows. This is so pervasive that we no longer notice how weird, how downright insane it is when contestants and presenters giggle, weep, swoon and generally effervesce to signal what a wonderful time they are having. Even the comedians have been infected; it may be me, but an awful lot of them seem to want to be nice, to be liked. Peter Cook should be their guiding light – his dandyish, withering, psychologically acute miseries are another great, bracing, pessimistic legacy of the postwar period.

Optimism is a pressure – it is stress-inducing and intelligence-lowering. Pessimism is a release: it is relaxing and mind-expanding. Read the Book of Ecclesiastes (“To every thing there is a season”) or Edward FitzGerald’s Rubaiyat of Omar Khayyam (“The Bird of Time has but a little way/To fly . . .”) to see how beautiful and peaceful zero expectations can be. And remember, when John Lennon wrote “It can’t get much worse” he was, I am sure, being ironic. Of course it can, it always can.

Bryan Appleyard’s recent books include “The Brain Is Wider Than the Sky: Why Simple Solutions Don’t Work in a Complex World” (Weidenfeld & Nicolson)