I was sitting gingerly on the bonnet of a classic Mercedes when I thought: “Hang on a minute. I know why I have been asked to write about this academic’s argument that we should all stop trivialising the midlife crisis.”

But I wouldn’t describe what happened to me as a midlife crisis. It’s true I was square in middle age – 42 – when I fell in love, fair and square, with a man who wasn’t my husband. I had to get divorced, and now I am, however you cut it, on my second marriage. There are as many people who would say to this, “Love is bullshit,” as would say, “The heart wants what it wants,” but if I know anything with certainty, it’s that you should never try to justify yourself to the first lot.

But there was a moment shortly before this that may also qualify as a midlife crisis: I went to Rotterdam to interview the pianist James Rhodes, who had just won a landmark case in the high court against his ex-wife. He had been abused as a child, suffered unimaginably poor mental health in early adulthood and then wrote a memoir that his wife had tried to block on the basis that it would be damaging to their son.

Legally, there were ramifications for privacy and freedom of expression; morally, it was a vindication of the supreme importance and right of a victim to tell their own story. But, in passing, he said: “How many people say: ‘Am I happy? No, not really. But I’ve got a mortgage.’”

Bryan Cranston as Walter White in Breaking Bad. Photograph: Sportsphoto Ltd/Allstar

For some reason, I had an almost physical reaction, my brain trying to flush the thought away with all the speed and noise of a train toilet. It wasn’t that I was unhappy. It was more the chronic wastefulness of never having asked: “Is this all there is?” just because it would have been inconvenient if the answer had come back: “No.”

This may be a working diagnostic tool for the midlife crisis, incidentally: listening too closely to strangers, or overthinking Seamus Heaney poetry (“The way we are living / Timorous or bold / Will have been our life”). But when the Canadian psychoanalyst Elliott Jaques coined the term “midlife crisis”, it was 1965 and the pressing social phenomenon that drove the discussion was marriage breakdown.

More than half a century later, Mark Jackson, a professor of medical humanities at the University of Exeter, notes: “That kind of compulsive infidelity, which occurred in men and women at around the age of 40, was regarded as the key determinant of that breakdown.”

In what was probably a familiar experience to the female academic of the mid-20th century, the anthropologist Margaret Mead had made this point about the midlife crisis, and gone beyond it, nearly 20 years before. In her book Male and Female, she floated this suggestion: we should allow two, three or four marriages. “The first,” Jackson summarises, “for youthful passion, your second marriage for parenthood, your third marriage for companionship.” (Jackson ends there. Maybe a fourth for different companionship, once you have had enough of the third?) “There’s nothing to suggest that can’t be all to the same person,” Jackson adds, although if that were true, the relationship wouldn’t have been in crisis in the first place.

Despite that original association between midlife crisis and infidelity, when we think of it now, we are more likely to think of motorbikes and other gaudy purchases; of Alan Partridge, Botox and self-delusion. We think of a crisis that isn’t real because, in the 70s and 80s, it became such a well-worn stereotype that it couldn’t help but be trivialised.

Jackson, one of Britain’s leading experts on ageing, says when you go straight from identifying a phenomenon to making a cliche of it, you never fully understand it. The period between adolescence and old age has been so disregarded by researchers, he adds, that the real mental and physical changes that can occur at midlife, and the way they interact with cultural norms and expectations, are not well understood. Yet the crisis that they can bring about can be hugely damaging to individuals and their families. So much so that Jackson is calling for a new medical speciality to cover the years between paediatrics and geriatrics, to help people navigate their middle years.

At a lecture, Life Begins at 40, at the Royal Society earlier this month, Jackson identified this cliche in its purest form: David Nobbs’ comic creation Reginald Perrin. “Forty-six years old. Bureaucrat. Behaves randomly. Aggressive memos. Tries to have an affair with his secretary. Collects childhood mementoes and burns them. Eradicates his past, eradicates his identity. Tries to disappear in some way, originally to kill himself, before he takes on a new identity.”

Leonard Rossiter and Sue Nicholls in The Fall and Rise of Reginald Perrin. Photograph: BBC Pictures Archives

“We took on this sense,” Jackson said, “that the midlife crisis was ridiculous and that it only applies to men. Both of those are wrong.”

The literature that launched the idea of a midlife crisis was, by the start of this century, based on very dated ideas about social norms. People in the 1950s and 60s got blasted by their mortality, futility and disappointment, at a time when they had already been in a marriage, and possibly the same job, for more than 20 years. Their child-rearing days were behind them, and they were assaulted by the oppressive emptiness of the nest and the shock of dying parents. Hilda Burke, a London-based psychotherapist and couples counsellor, suggests that the loss of one’s parents is about more than just grief: “You’re now at the frontline of life’s battle [against death], whereas before there were generations ahead of you.”

Yet today, we are just as likely to be facing the pell-mell demands of the sandwich generation (pressures from ageing parents and small children) or children who can’t afford to leave home, than worrying about an empty nest. I couldn’t name any lay authority who has thought harder about relationships than the sex advice columnist Dan Savage; he thinks the midlife crisis has slipped out of the cultural lexicon because it is no longer relevant. “People used to settle down without ever having had a life. Now people have a life. They marry when they’re 33, 35. They have got to know themselves. They don’t wake up when they’re 40 and say: ‘What the fuck? I never got to do anything that I wanted to do – I’m just doing what I was supposed to do,’ and blow up their own lives and those of their children and spouses.”

Pausing to consider that cultural erasure – and why you rarely see that character in films and sitcoms any more, the volatile, thwarted individual who makes everyone roll their eyes and mouth “midlife crisis” – it may be that, for a while, ageing simply didn’t exist for the purposes of drama. A 60-year-old man could be an action hero; a 45-year-old woman could be a romantic lead. Perhaps in the new sophistication of the box set, this is going into reverse: Walter White is having a midlife crisis; the Americans are having the lesser-spotted dual-marital midlife crisis.

If the typical discussion of the midlife crisis lampooned men, it largely ignored women, putting it all down to the menopause. This rather wishful reading consigned women to the biological margins and made us into a footnote to the male crisis, whose drive for a younger model was explained away by midlife women becoming sexually obsolete. But, just because the parameters of the discussion about midlife crises were obsolete, it shouldn’t have blinded us to the kernel of truth: that life is finite, that sensations of waste, futility, emptiness and inauthenticity, are a natural consequence of realising that.

Steve Taylor, an author and lecturer at Leeds Beckett University, who has written about how to fight that concertina effect, where the older you are, the faster time moves, says: “I think ultimately it’s a crisis of meaning. The values of our culture are largely based on achievement and on consumerism, and if you follow those values to their logical end, you may become a successful person, you may be financially settled. But you reach the age of 40 or 50 and think: what is the point of all this? You haven’t actually begun to live authentically.”

The first half of life is about external things: a career and building prosperity. Midlife is more inward-looking

Taylor describes this dichotomy, but notes that authenticity doesn’t come free. “I did some research on people who had recovered from cancer. I asked: how did they live differently in the light of it? One woman decided to live for herself. She went paragliding, she went travelling on her own. The people in her life thought she had gone crazy.”

It’s not just that there is a vanishingly thin line between authenticity and selfishness, because – especially in a family – there is no such thing as consequences-for-one. But it is also “like a prison break”, Taylor says. “The people around think: ‘Wow, what have I been doing? Couldn’t I escape? Why didn’t they take me?’”

These urges are not necessarily destructive. Burke raises the Jungian perspective that “the first half of life is about external things, trying to build a career, build prosperity. Midlife is much more inward-looking.” Jung describes “letting go of the overdominant ego to contemplate the deeper significance of human existence”. Which doesn’t necessarily mean paragliding or getting divorced – although one should never underestimate the challenges of trying to live with someone while they self-actualise. You wouldn’t necessarily spot a person having a Jungian midlife crisis by “what she is wearing or what jewellery she has bought”, says Burke. It would be more likely to centre on finding a meaning in your career, or rather, losing patience with its meaninglessness.

Lucia Knight, author of X Change (How to Torch Your Work Treadmill), was spurred to a career change from a swamp of dissatisfaction: “My body didn’t feel right, I didn’t look the way I wanted to look, I didn’t feel as happy as I used to feel. But more than that, I had this sharpening feeling that I was running out of time,” she says.

That sense of physical decay has profound connotations, literal and metaphorical. Your body reflects your own mortality back at you. We fixate on cyclical, regenerative milestones – changing seasons, new terms – then get this harsh reminder in middle age that we are not regenerating at all; we are hurtling downhill. Yet this is the element of middle age that is always the most ridiculed – the mankini, the weekend warrior, the mutton dressed as lamb. We are denounced for the shallowness of our vanity, when what we are feeling is the sudden, vertiginous realisation that physical decay isn’t trivial. It has meaning. It means we’re gonna die.

Zoe Williams … ‘We may have made a decision to bury a universal life event under ridicule because the alternative is inconvenient.’ Photograph: David Levene/The Guardian (Car courtesy of the Classic Car Club)

Then there is the midlife career crisis – a whole other caper. People always, Knight says, talk about retraining. “They want to escape from work: it doesn’t fit any more. It becomes more and more obvious, ‘I’m not doing work that matters.’” (This may – of course – be a running theme for everyone, and is sometimes triggered by work itself, which can be circular and pointless in subtle ways that takes 20 years to work out.)

Feelings of futility at work are much easier to talk about, though, than the same feelings at home. And they can be easier to fix – even if Knight urges some caution. “People can drop huge amounts of money doing a master’s, and they often do report that it has made them happier. We all tend to want to see a situation in its best possible light.” He encourages anyone tempted to quit their job to take “experimental days off”, perhaps shadowing someone else. “You can learn so much and it’s not so seismic.”

Perhaps there is another reason that we identified this difficult age, then immediately made it into a joke, narrowed its perimeters and deliberately constrained our understanding so that it could be consigned to history: if everyone took their midlife crises seriously, what would that make our society look like? If we all dropped everything to go on a voyage of self-discovery, that would be … well, that would be awful. We may have made a collective decision to bury a universal life event under ridicule because the alternative is monumentally inconvenient.

Jackson is “torn”, he says. It can be very liberating; it can be very destructive. A truth that has been painstakingly established with cancer and with mental health is catching up with middle-aged people: whatever the problem is, “shame and stigma don’t help”.