By tradition, this is the season for personal reinvention, but these days it’s hard not to feel cynical about the idea of a triumphant liberation from the past. In the news, Brexit provides an hourly reminder that merely wishing to bring about a glorious fresh start is no guarantee that calamity won’t be the result. Meanwhile, other dark developments – from the erosion of American democracy and the resurgence of the European far right, all the way to climate change – fuel a sense of foreboding that isn’t exactly motivational when it comes to self-improvement: the creeping fear that you might be living in the end times is a poor basis for making a new beginning. In any case, the never-ending debate on nature versus nurture seems to be drifting toward a gloomy acceptance that there’s much about ourselves we’ll never change. “DNA isn’t all that matters,” writes the geneticist Robert Plomin, whose book Blueprint epitomised this mood last year, “but it matters more than everything else put together in terms of the stable psychological traits that make us who we are.”

Then again, the self-reinvention narrative was always a bit suspect to begin with. For one thing, it’s by no means clear that it’s possible to transform yourself through the simple application of individual willpower: wherever you come down on nature and nurture, it’s undeniable that we owe much of our success or failure in life to our circumstances, and to luck. Then there is the infuriating psychological quirk of “hedonic adaptation”, otherwise known as the happiness treadmill. Succeed in improving your life, and the improvement will soon become part of the backdrop of your days, and thus cease delivering pleasure; to recover that sense of vitality and zest, you’ll have to reinvent yourself again, ad infinitum.

Finally, there’s the conundrum that the self being reinvented is the same one that’s doing the reinventing – so your existing flaws invariably get baked into your vision of the future. If you yearn to become, say, more productive or empathetic or physically fit in 2019, how do you know that very yearning isn’t just another expression of your tendency to beat yourself up, which you’d be better off addressing than indulging? Or suppose you plan to conquer your perfectionism: how will you avoid getting all perfectionistic about that? There are no completely fresh starts, there is no year zero. You’re already hopelessly ensnared in the only life you’ll ever get.

In response to the prevailing mood, there has been a noticeable change of tone in the world of self-help, a publishing genre historically dedicated to promising massive, near-effortless transformation overnight, or in a couple of weeks at most. For a while now, that hyperbole has been losing ground to a spirit of anti-utopianism – of accepting yourself as you are, building a good-enough life, or just protecting yourself from the worst of the world outside. Adult colouring books are the most easily mockable manifestation of this urge. But it’s detectible, too, in the unceasing stream of Scandinavian lifestyle concepts – hygge, lagom, and the rest – with their focus on hunkering down and getting cosy; and in the ongoing rediscovery of Stoic philosophy and the championing of “resilience” as techniques for enduring life’s blows.

And it’s everywhere present, this time around, in the phase of the publishing calendar irritatingly known as “New Year, New You”. One thought-provoking example is a new edition of Solve for Happy by Mo Gawdat, formerly a senior executive with Google X, the search giant’s secretive research and development arm. Generally speaking, the notion that “happiness is an engineering problem” is one to distrust. But Gawdat, far from championing the tech multimillionaire lifestyle as the only one worth aiming for, writes movingly of having achieved it, only to discover its emptiness. And he has endured far worse, losing his 21-year-old son Ali as a result of complications during routine surgery.

The hyperbole of overnight transformation has lost ground to a spirit of anti-utopianism – of accepting yourself as you are

At the core of Gawdat’s “formula for happiness” is the venerable observation that happiness equals reality minus expectations: in order to feel distress because your life is lacking something, you must first have had some expectation of attaining that thing. (My life lacks a 70ft yacht, but this causes me no suffering, because I never imagined I’d have one.) The argument is not, as progressive critics of self-help sometimes imagine, that disadvantaged people need only stop expecting anything better in order to be content. Some expectations – a reasonable standard of living, healthcare, fulfilling work, social connection – may be entirely rational. But seeing the truth of the formula acts as a kind of sieve, allowing you to separate the handful of things you genuinely want from life from those you’ve been socialised into believing you should want. The latter aren’t worth the pursuit – and if they are the reason you’re trying to invent a “new you”, you’re better off sticking with the old one.

¶

One of the most rigorous articulations of the new mood of acceptance is Happy Ever After: Escaping the Myth of the Perfect Life by Paul Dolan, a professor of behavioural science at the LSE and, the publicity material explains, “an internationally renowned expert in human behaviour and happiness”. His book is a persuasive demolition of many of our cultural stories about how we ought to live, including the idea that there’s anything particularly desirable about being a senior academic or a renowned expert. In fact, his data suggests, pursuing education beyond the age of 18 is unlikely to make much positive difference to the pleasure or sense of purpose you experience in life: on average, after secondary school, “happiness decreases as education increases”.

Paul Dolan identifies the ‘narrative traps’ in wellbeing research. Photograph: Jeremy Baile

As Dolan concedes, it can be notoriously hard to pin down the direction of causation in wellbeing research: it could be that gloomier people are more prone to doing university degrees, rather than that degrees make people gloomy. Yet either way, the belief that more education equals more fulfilment is a clear example of what he calls a “narrative trap” – a socially imposed message about the ideal life that doesn’t match real experience. This ideal often ends up doing more harm than good, either by propelling people into lives they don’t enjoy, or by wrongly convincing those who don’t make the grade that they are missing out on a more satisfying existence. Another trap is the belief that higher-status jobs reliably bring more satisfaction (in fact, florists are generally much happier than lawyers), or that a larger income necessarily buys more happiness (it does, but only up to about £50,000 a year; beyond that, tasks related to earning more money squeeze out more enjoyable ones).

These sorts of findings are increasingly well known, but where Dolan excels is in drawing attention to how stubbornly we resist their implications. If happiness and a sense of purpose are your goals in life, then a “good job” or education or salary that fails to deliver them isn’t really “good” in any meaningful sense of the word – which makes it a strange thing to strive for, or to encourage your children to strive for. Oh, and speaking of children, the evidence is that parenthood won’t make you happier, either. (It does boost people’s sense of purpose, although apparently not more than various other things.) Likewise the dedicated pursuit of physical fitness, which turns out to lead to less happiness than you’d think. And marriage: it’s true that married people tell researchers they are happier than when they were single, but only if their husband or wife is present in the room during the interview.

The message of self-reinvention is never relax, since you could always benefit from acquiring more money and status

What makes Happy Ever After somewhat radical, at least by the standards of popular psychology, is its recognition that these narrative traps aren’t simply inexplicable mistakes we happen to make, but the products of ideology. They may not serve us, but they certainly serve the system in which we find ourselves embedded. The pursuit of wealth or social mobility might not bring happiness, but it does fuel economic growth – while marriage, parenthood, fitness and the rest keep the whole operation running smoothly into the next generation. Dolan focuses on how uniquely detrimental such messages can be for children from working-class families. Stereotypes about appropriate accents and lifestyles may deter them from going to university at all; those who make it into middle-class professions then face self-consciousness and insecurity about fitting in. Dolan, raised “lower working class” in east London, writes that he still struggles with the cultural codes of academia: “I weight train with bodybuilders. Seeing a blazer or a pair of loafers at a bodybuilding competition is as rare as rocking-horse shit.”

The new crop of anti-perfectionist self-help books are an important counterweight to the conventional message of self-reinvention, which is that there’s no point at which it makes sense to be satisfied with your situation and finally relax, since you could always benefit from acquiring more money, status, education, and so on. What’s less clear is whether this humbler kind of advice is any easier to implement, on a practical level, than the old sort. Apart from anything else, our narratives about the perfect life aren’t just beliefs we can choose to jettison by a mere act of will, after reading about research that refutes them. They are deeply entrenched in the culture, reinforced by the media, inculcated in us as small children, not to mention in our genes. (There are some obvious evolutionary advantages to constantly craving more resources, and never feeling as if what you’ve got is sufficient.) Moreover, no research finding about the average happiness of the general population can decisively prove that a given lifestyle choice is the right or wrong one for you, with all your idiosyncrasies. One chapter in Happy Ever After gamely makes the case for polyamorous relationships as a path to increased happiness, but whatever your reaction to that prospect – thrilling erotic adventure, or indescribable hassle? – it’s not clear that you should try to override it based on the results of academic studies.

There’s another, more mind-bending problem with using this kind of research to direct personal change, which is that many such transitions are what the philosopher LA Paul calls “transformative experiences”: they turn you into a person so different that you’re unable, from the vantage point of the present, to imagine what that future person will make of them. To pick the most obvious example, becoming a parent might transform you into the kind of person who adores having children, even if beforehand you weren’t. But it might just as easily work the other way, turning someone enthusiastic about the prospect into the kind who’d never have chosen to do so.

Nonetheless, it’s psychologically freeing to be reminded that there is no single path to satisfaction – and that if circumstances or personal preferences disbar you from following the herd, you still have a good shot at fulfilment. “Quite a lot of what passes itself off as dialogue about our society,” the essayist Tim Kreider has written, “consists of people trying to justify their choices as the only right or natural ones by denouncing others’ as selfish or pathological or wrong. So it’s easy to overlook that hidden behind all this smug certainty is a poignant insecurity, and the naked 3am terror of regret.” Much of the time, when it comes to building a meaningful life, you’re flying blind. But the comforting truth is that so is everyone else.

¶

Then again, judging by its continued dominance of the self-help shelves, you’d be forgiven for concluding that the key to a perfect life was indisputable: lots and lots of Buddhist (or at least Buddhist-inspired) meditation. This is ironic, since Buddhism embodies one of the earliest confrontations with the truth about the perfectionist standards by which we judge the world and ourselves – that this is a recipe for permanent dissatisfaction. The basic situation, Buddha famously said, is that life is suffering. Everything is impermanent; old age, sickness and death are our inescapable human fate. And your philosophy of happiness had better acknowledge these realities, otherwise the only possible result is even more suffering, for you and everyone around you.

Especially eloquent on life’s smaller dissatisfactions … South Korean Zen writer Haemin Sunim. Photograph: WENN Rights Ltd/Alamy

While he probably wouldn’t put it so bluntly, this is the spirit that imbues a new work by the South Korean Zen writer and former monk Haemin Sunim, Love for Imperfect Things: How to Be Kind and Forgiving Toward Yourself and Others. In these snark-saturated times, it’s cheering that a voice as quietly friendly as Haemin’s can make you a mega-celebrity: he has a combined social media following of around 2 million people, plus a previous global bestseller, The Things You Can See Only When You Slow Down. (In Seoul, where he lives, he presides over a Zen-infused therapy centre, the School for Broken Hearts, but the primary vehicle for his teaching is Twitter.)

Haemin is especially eloquent on life’s smaller dissatisfactions, and how they can sometimes be trickier to deal with than the bigger, more dramatic ones. For example, though it’s a good thing that we talk so much more openly today about mental illness, one perverse consequence is that it can actually be easier to admit to a serious depression than to a milder, pervasive sense of disappointment in life. “Unlike other emotions, disappointment is very tricky to express: it comes out as petty and small-minded,” Haemin writes; it also tends to sound like you’re blaming other people for failing to measure up. Yet of course it’s a far more widespread problem than severe suffering. It has been argued that Buddha’s observation that “life is suffering” might be more accurately translated as something like “life is bothersome”. (With luck, extreme agony will be very infrequent in your life, but a background sense of things being not quite right may be truly close to universal.) The first step towards relieving this kind of discontent, Haemin suggests, is to recognise the untenability of the demand that you, or anyone you encounter, should demonstrate perfection to begin with. Much of the bothersomeness of daily life arises not from circumstances themselves, but from the insistence that they ought to be other than they are.

Having not yet attained Haemin’s tolerance for other people’s flaws, I can’t resist observing that his wisdom all too often comes across as platitudinous. The book’s prose passages are interspersed with sections laid out as blank verse, inadvertently demonstrating that mundane reflections aren’t transformed into profundities merely by centring them on the page and inserting a few line breaks. (A typical example: “If someone did not ask for your help, / do not try to solve her problem for her. / Though your intentions may be good, / You risk taking control away from her / and injuring her self-esteem.”) Still, he’s not wrong. And behind the sporadic banality lurks a bracingly hard-headed world view: reality is what it is, and a lot of unnecessary misery arises from demanding that things shouldn’t be the way that – as a matter of stubborn fact – they are. This is not a counsel of resignation; having accepted the reality of your situation, it may well be appropriate to try to change it. But not denying how things stand is the essential first step. Or, as the psychotherapist Carl Rogers put it: “The curious paradox is that when I accept myself just as I am, then I can change.”

It can be easier to locate fulfilment in the future, rather than run the risks involved in trying to achieve it now

Ironically, if not very surprisingly, the wellbeing industry has proved adept at turning this new spirit of modesty and acceptance into another expensive consumerist pursuit. Those Scandinavian “secrets of happiness” are a case in point: “hygge” may evoke contented relaxation around a familiar fireplace with old friends, but that doesn’t mean you can’t spend almost £200 on a specially hygge-appropriate plant-dyed pillow, or £80 on a set of candle holders, at the website hyggelife.com. (And while “lagom” may mean “just the right amount” in Swedish, there are at least six recent books in English on the subject, which isn’t just the right amount, but too many.) Your effort to become the sort of person who finds happiness in what they already have can easily become its own interminable quest, in which success – and therefore happiness – always lies in some fantasy of the future, rather than in the here and now.

As always, this is capitalism’s fault. But most of us are complicit: we chase unattainable fantasies of self‑reinvention, rather than confronting reality, at least in part because life is easier that way. This is one of the lessons of an absorbing recent addition to the anti-perfectionist self-help subgenre, The Courage to Be Disliked, by Ichiro Kishimi and Fumitake Koga, published in English last year. (By then, it had reportedly already sold more than 3.5m copies in the original Japanese and other translations.) Despite all the “new Japanese phenomenon” marketing – the book was described by one critic as “Marie Kondo, but for your brain” – it is primarily an accessible exploration of the work of the Austrian psychotherapist Alfred Adler. He held that we frequently cling to our problems, no matter how much we complain about them and claim we want to eradicate them, because overcoming them necessitates an encounter with fear. It can be easier to locate fulfilment – and fulfilment in intimate relationships above all – in the future, where we never quite have to do what it takes to attain it, rather than run the interpersonal risks involved in trying to achieve it now.

The problem, as Kishimi and Koga make clear, is that this only makes for more suffering in the present, by systematically biasing you towards taking the kind of actions that postpone, rather than build, a meaningful life. In this way, fantasies of total self-transformation don’t simply fail, they also block change of the more modest – but real – kind. And in any case, the future never seems to arrive: the truth is that the present is the only time it’ll ever be possible to make a change. Transformative self-reinvention may be an overoptimistic dream, but defeatism about change is its own kind of false comfort, too: both are forms of absolutism that serve to justify passivity. We will fail to reinvent ourselves this January, or next month, or next January, or ever. But once we finally get that fact into our heads, we might at last be able to start making a few improvements.