One day last year, Liam Williams locked himself out and tried to climb in through his bedroom window. “I’d done it before very skilfully when drunk,” he says, “but this time I was hungover, so I guess I had that reduced inhibition, but not that derring-do – you know, the reckless optimism of a drunkard.” It didn’t end well. “It was only the first storey but I didn’t have any shoes on and it was quite a high window. I fell and broke my heels. It really hurt.”

The comedian is telling this story to psychotherapist Philippa Perry and me as we meet in a London cafe to consider the merits of optimism and pessimism. Is pessimism necessarily bad for you? What health benefits come with being optimistic? Does being optimistic help you in relationships? Does being pessimistic make you pragmatic about a prospective lover’s shortcomings? If you’re as bleakly pessimistic as Eeyore, can you change? If you’re as misguidedly optimistic as Mr Micawber, can you get a firmer grip on reality? More troublingly, what looks like pessimism to one can seem like optimism to another. Consider Williams’s attempted break-in. Perry suggests that his climb was optimistic. Liam worries it was doomed by pessimism. “It comes under the heading of risk-taking,” says Perry. “Optimists are more likely to take risks – they think they can drive into that gap in traffic or climb through windows.” She pauses before adding: “That’s not necessarily a good thing.”

“I was a bit desperate,” says Williams, “so I made a ridiculous decision. As I was climbing, I lacked that crucial optimism that I was going to get in. I was physically impaired and I probably had a narrative in my head that this wasn’t going to go well, which I wouldn’t have had when I was drunk.”

“That,” says Perry, “is the self-fulfilling prophecy thing.” What does that mean? “If I go to a party expecting to have a good time, look people in the eye, I may well do. If I walk in hang-dog thinking nobody wants to speak to me, I probably won’t.” Viewed thus, Williams expected to fall. Perhaps he even wanted to.

Soon after the fall, a friend asked Williams how he’d been feeling at the time. “I said maybe a bit depressed. And she said: ‘Maybe you took a reckless decision because you were depressed.’” Williams compares this to the EU referendum: “Maybe it was like Brexit – ‘My life is so shit at the moment that I can take this grave, reckless decision and at least something will be different.’” The year 2016 was, he thinks, marked by such pessimism. “Those who voted for Brexit and Trump were so pessimistic about how things are going, they felt any change must be better.”

“Which,” adds Perry insightfully, “is quite optimistic.” Good point: to their opponents, such as Williams, Brexiteers and Trumptonians may have looked like pessimists who, in a fit of infantile petulance, were destroying their respective polities. But, seen another way, they were cock-eyed optimists who believed change was possible.

Williams tells us he has turned away from standup, in part because he feared he was making his audiences as politically apathetic and pessimistic as he was. In 2014, he did a show in Edinburgh called Capitalism, which (despite the Guardian’s five-star review) made him queasy. His routine revolved around mocking his lack of political fibre: the twentysomething Cambridge English graduate didn’t have what it took to go on protests or join Occupy. Nor did he have the intellectual drive to develop a sophisticated critique of capitalism by reading Thomas Piketty or Slavoj Žižek, still less act on it.

“I kept thinking: ‘There’s a lot wrong with the world, and crises are getting worse, but I can’t change any of it,” he says. Lots of people feel that way, I suggest, especially twentysomethings, and their impotence then slides into pessimism. “That’s right. But I detected in myself an apathy, a willingness to focus on trivialities,” he replies. “I wouldn’t do a show like that again, because those who saw it might walk away having let themselves off the hook.” Maybe our most evisceratingly pessimistic comedians – think Frankie Boyle or Stewart Lee – let us off the hook in the same way. Perhaps (and this is just my own pessimistic thought) comedy is a successful business model in austerity Britain because it provides alibis for inaction – and, as a result, makes things worse.

So Williams has turned his back on standup. Instead, he has co-written and performs in what he calls a panto for grown-ups, called Ricky Whittington and His Cat. “He turns up in London full of optimism,” says Liam, “and finds it hostile and plagued with rats.” Still is, I point out. “Then he leaves in despair before hearing voices calling him back. He projects his inner voice on to the bells. They call him back.”

Perry says this is not unusual: “He can’t own his optimism, but projects it on to the bells.” True, but on the plus side, he overcomes his pessimism, returning to London, becoming mayor and building state-of-the-art cycle lanes or whatever it was he did.

‘Pessimists see the worst in people – that’s why they’re more interesting’ … Philippa Perry. Photograph: David Levene/The Guardian

So, if Williams is anything to go by, pessimism is bad for you. But it’s more complicated than that, says Perry. She hands me a list of pros and cons for both outlooks. Pros of optimism include: better mental and physical health, quicker recovery rates after illnesses and operations, lower stress, longer life, higher trust in others and thus better relationships. Pessimism’s pros include: better at budgeting, better insured, on time. Interesting: both she and Williams arrived on time, suggesting that they’re both pessimists. “Pessimists are always punctual,” says Perry. “Optimists think the bus is going to be on time, that they can get somewhere quicker than they can.” I’m hating optimists already.

But surely, even if you’re an optimist, it’s better to be with a pessimist? At least they’ll be on time. And they’re better company. “Pessimists are much more cynical,” says Perry. “They see the worst in people – that’s why they’re more interesting. Though you wouldn’t want to be on a desert island with one.”

Nonetheless, recent research suggests that pessimism is actually bad for you. A 2011 study, published in Stroke, a journal of the American Heart Association, measured respondents on a 16-point optimism scale and found that, for every point increase, there was a corresponding 9% decrease in acute strokes over the two-year follow-up period. But why? “Our work suggests that people who expect the best things in life actively take steps to promote health,” says Eric Kim, who has researched the subject for the Harvard TH Chan School of Public Health.

But this was called into question in early 2016 by Helsinki researchers who studied middle-aged Finns and found that, while pessimism was linked to an increased risk of death from coronary heart disease, there was no indication that optimism led to a decreased risk.

Both studies have one thing in common: they suppose optimism and pessimism to be measurable. Levels of pessimism were measured in Helsinki by asking people to respond to such gloomy statements as: “If something can go wrong for me, it will.” Optimism, meanwhile, had such choice lines as: “In uncertain times, I usually expect the best.”

None of this, you’d hope, means it’s easy to pigeonhole respondents. “Maybe labelling someone either way is as reductive as saying they’re an extrovert or introvert,” says Williams. Good point – yet scarcely a month goes by without another study telling us that optimists are healthier.

Here’s another problem. What if pessimism is something you’re doomed to by genes and environment? A 2006 study suggested that outlooks are forged in childhood. Those with high socioeconomic status, unsurprisingly, were more optimistic. More interesting was the idea that pessimism tended to stick even in the face of socioeconomic fluidity. So if you had a poor, miserable childhood, you might well be pessimistic even as you bask on your parvenu yacht off Corfu.

“That’s not to say you’re hard-wired to either,” says Perry. “You can change.” So we can cultivate optimism? How? Kim suggests keeping a log for a fortnight of any kind things you do for others, to achieve your “best possible self”, as well as attending mindfulness courses, getting into therapy, writing down three things each day you are grateful for. Then, at the end, you see how you feel.

There is, says Perry, a neurophysiological basis to the possibility of becoming optimistic: the brain’s left inferior frontal gyrus fires up with good news; the right inferior with bad news. But if your life is marked by bad news, that left inferior is apt not to register good news. “Most people are optimists, and the more optimistic you are, the less likely the right inferior is to respond to negative information,” she says.

Here’s an interesting problem with optimism. You can have too much of it. Neuroscientist Tali Sharot, author of The Optimism Bias, argues that 80% of us suffer from the illusion that supplies her book’s title. “The idea,” says Perry, “is that the optimist knows smoking kills but thinks: ‘It just won’t kill me.’” Or, as Sharot puts it: “We underestimate our likelihood of suffering from cancer, or being in a car accident. We overestimate our longevity, our career prospects. In short, we’re more optimistic than realistic, but we are oblivious to the fact.”

Sounds like we should cultivate pessimism then. I’ve brought along Arthur Schopenhauer’s On the Suffering of the World, in which the German pessimist describes the nature of human life thus: “We begin in the madness of carnal desire and transport of voluptuousness, we end in the dissolution of all our parts and the musty stench of corpses … Does it not look as if existence were an error the consequences of which grow more and more manifest?” Though I place the book prominently on the cafe table, neither comedian nor psychotherapist is tempted to pick it up.

“Since that fall, my life has been better,” says Williams. “I’ve pulled myself together and it made me reflect.” “So did you get another key cut?” asks Philippa. “I gave one to the neighbours,” he replies. “You’re marvellous!” she says chirpily. “That’s learning to trust people.”

Williams fills his teacup – his half-empty teacup – and looks doubtful. “But there’s a degree of pessimism in that, right?” he says. “Because I know I’m going to forget the keys again.”