There was once a banker in his 50s who had worked seven days a week for 25 years and become a very wealthy man. Then, at the apex of his career, he looked around him and realised that he had entirely neglected his family; as a result, his family had rejected him. The regret was overwhelming, and came out in panic attacks every Sunday. Would this man be able to find a way out of this cruel place he had created for himself?

This man was a patient of the psychoanalyst David Morgan, of the Institute of Psychoanalysis, who spent several years helping him explore what had compelled him to work so hard and to ignore his children (he has been anonymised and gave Morgan permission to use his case). It became clear that this need to become richer than everyone else had roots in his very early childhood, when he watched his parents nearly starve to death during the 1980s miners’ strike. He had, unconsciously, repeated this by impoverishing his children by not being there for them, in turn impoverishing himself of these loving relationships, in his efforts to overcome the traumatic poverty of his childhood.

“That complicated understanding,” Morgan explains, “freed things up, setting his regret in a generational context so he didn’t have to feel quite so guilty for acting something out, because it was beyond his ken. It doesn’t mean that he can’t feel real pain, but that pain is given a sense of history.” This meant his regret could be understood and given meaning – and that changed his life.

Regret can be all-consuming, and it can destroy lives. We can see it all around us, whether it is the man who cannot forgive himself for cheating on his first girlfriend and has not had a serious relationship in 30 years. Or the woman who is so tied up in wishing she’d had a child with her ex-partner, instead of breaking up with him, that she cannot find happiness in her current circumstances.

It is not unusual for patients to seek therapy because they feel plagued by regret and unable to live full lives because of it, says Morgan, whether it is over affairs, career choices or relationships. The kind of regret that brings people to his consulting room is “paranoid and persecutory. It’s: ‘Oh God, I’m so terrible, I’m dreadful,’” he says. It is self-flagellation, and it can be incredibly damaging to our mental health. It is exhausting, it sucks all joy and fulfilment from our days and it leaves us stuck, always looking backwards and unable to move forward in our lives.

The cognitive behaviour therapist Windy Dryden says that, when we are trapped in this cycle of regret, characterised by rigidity and inflexibility, we only seem able to blame ourselves for what has happened, rather than seeing our behaviour in a wider context and understanding why we took the path we did based on the information we had at the time. Under these conditions, regret will become toxic.

Yet, strange as it sounds, there are people for whom this kind of regret can become a safe haven, because it can protect them from the pain and risks of living a full life. Catriona Wrottesley, a couples psychoanalytic psychotherapist at Tavistock Relationships London, says that regret can be used by some as “a defence against loving”. She describes a scenario, made up of various anonymous patients: a woman, whom I’ll call Amy, after leaving a long-term marriage, held on to her regret at having married too young and stayed too long, and was determined not to make any mistakes the next time around. Ready to make a fresh start, she signed up on various dating websites, and began going on first dates. Although there were men who wanted a second date, Wrottesley explains: “There was always something about them she felt unsure about – somebody’s shyness, or a look in his eye. She was very preoccupied with getting into the right relationship but, unconsciously, she was doing all she could to protect herself from getting into one at all, because she was terrified of repeating the disappointment and the hurt she had already endured.”

Amy was in danger of falling into another trap outlined by Dryden: if you avoid doing anything that you might regret later, you will disengage from relationships, opportunities and eventually life itself – and the irony is, there is no more powerful source of regret than that.

Once Amy could make a shift towards allowing herself to get it wrong, she was able to move beyond the first date with a man, even though she was not sure he was entirely right for her – this was the only way she could get to know which men she liked and which she did not. We have to open ourselves up to the possibility of making mistakes and regretting them, in order to learn from the experience.

“That’s not an easy thing to do,” Wrottesley says, “but with practice, it does get easier, because the more we can allow ourselves to make mistakes, if we can learn from them, the fewer mistakes we make.” She has seen patients like Amy go on to develop long-term, fulfilling and loving relationships.

But regret does not only serve as a defence against the risk of loving – it can serve a darker purpose, allowing people to hide from the deeper pain of remorse. Morgan says: “Remorse involves insight into what one has done to others. That is the beginning of becoming aware of how one behaves and wanting to do something differently. It is a real breakthrough in therapy when people can begin to experience genuine remorse for what they’ve done. Something authentic starts to happen.”

What does it take to move from using regret as a stick with which to beat ourselves to experiencing remorse as a way forward to a better future? Dryden believes it requires a shift from an inflexible mindset filled with certainties such as: “I absolutely should have done this” and: “I absolutely shouldn’t have done that”, which he calls “the enemy of learning”, to asking the question: “I wonder why I didn’t do that?” Once you are occupying this more flexible frame of mind, he suggests imagining you are talking to a loved one, be it a child, friend or spouse, and to find that same space of acceptance and compassion for yourself.

He makes a point that I find myself thinking about weeks later: “There is a tendency with regret to see the pathway you didn’t take as inevitably better than the pathway you did.” It may well be that this other pathway would indeed have worked out better – but the point is that we cannot know for sure. It is that certainty, that transformation into knowledge of what can only ever really be a supposition, that is the hallmark of toxic regret. It is the ability to accept yourself, to recognise that there was a wider context to your actions and to understand that you made the decisions you made based on the values and the information you had at the time, that leads to remorse and self-knowledge. Dryden says: “Take the psychological equivalent of cod liver oil, which doesn’t taste nice but will do you good: accept the point, difficult to swallow though it may be, that yes, it would have been nice if you had made a different choice, but you could only have acted as you did at that time in those circumstances.”

For some people – and for some regrets – Dryden says this process can be swift: he specialises in single-session therapy, where he sees clients only once to help them overcome a specific problem. For other people and other regrets, the process can take much longer. Carine Minne is a consultant psychiatrist in forensic psychotherapy and a psychoanalyst, working in the Portman Clinic, at the Tavistock and Portman NHS Trust and in a high security hospital with disturbed patients, some of whom have committed violent crimes. One important part of her work, she explains, involves addressing the devastating trauma of their childhoods, as well as the horrors they have committed that have brought them into forensic psychotherapy. That, eventually, will involve facing up to regret.

“One of the things I try to do with these kinds of patients is to help them develop an awareness of who they are and what they have done,” she says. “Regret comes in a spectrum” – at one end, there is regret for others; at the other, there is “self-regret”. This is where many of her patients start out: some regret being caught, a lot regret having been transferred to the high-security hospital because it is better to be seen (and to see oneself) as a criminal than as mentally unwell. But the hope is that over the long course of treatment – between five and 10 years or more for her most disturbed patients – she can repair some of the psychological damage from neglect and abuse in their early lives, and their regret can become focused on others rather than the self.

This kind of meaningful regret for others, she says, is “a tremendous achievement, but it takes a long time before the mental structure, the scaffolding of the mind, is sufficiently solid to be able to experience it.” When I ask what that looks like, she replies: “It gives me goose pimples thinking about that question, because I’ve had men ending up in floods of tears. I remember one man, who had never cried in years of therapy, staring at me with watery eyes and saying: ‘If I start, I know it’s never going to stop, because there is an ocean of tears to come.’” Remorse, she says, “is one of the most sophisticated experiences that someone can possibly have. That is why I’m always astonished when a judge, at the end of a criminal trial, says to one of my potential patients: ‘And what’s more, you have not shown any remorse!’ If that person in the dock had the capacity to experience remorse – well, they never would have done what they did.”

Hearing these words, it is impossible not to realise the danger of the saying “No regrets”. Being able to feel regret – the right kind of regret, which can be understood, worked through and can lead to remorse and repair – is the strongest sign of a life meaningfully lived, of a healthy mind. “If you don’t feel regret,” Wrottesley explains, “and you’re without remorse, you will find yourself in the very difficult position of continuing to do something destructive without insight, causing damage to family and friends.” For her, “regret, though it’s very painful, can be a gift. It can be the doorway to a better way of living, of being with others.”

Dryden agrees: “People who say ‘I regret nothing’ are either saints or stupid, in my view. Regret based on flexible attitudes is the hallmark of mental health. It is a sign that you are engaged with life.” Without regret, we cannot learn from our mistakes, and we are destined to repeat them, as with Morgan’s banker. The panic attacks and regret that brought him to therapy, Morgan says, “were a message from his soul saying something was wrong. That was the healthiest part of him.” He is proof that, as Morgan says: “There is life after regret. One can recover.”

How to regret

Advice from Catriona Wrottesley, couples psychoanalytic psychotherapist at Tavistock Relationships London

Accept there is no solution for feelings of pain, loss and disappointment. They are part of being alive, and can be experienced and survived. Engage with life in all its ups and downs.

View regret as an opportunity to do things differently next time, rather than a signal that you should give up trying altogether.

If you have caused hurt or harm, instead of beating yourself up, do what you can to repair the damage.

Support friends and family through emotional challenges, and allow yourself to be supported, too.

Allow yourself to “get it wrong”.