Imagine a virus we don't fully understand is killing young men in record numbers. It kills three times as many British men as women, although nothing adequately explains why. The government confirms that while almost all other leading causes of death are being slowly eroded by medical and social progress, deaths caused by this virus are at their highest for decades. Yet the money we spend on researching and treating the problem stands at a fraction of what we spend on those other leading causes of death, as do charitable donations from the public.

The deaths this virus has caused have risen to the point that if you are a man between 20 and 49, you're more likely to die from it than cancer, road accidents or heart disease. It is the biggest threat we face, the number one killer. The figures are chilling, and yet still it is something we hardly discuss, in public or private — which is a significant part of the problem itself.

Last February, the Office for National Statistics published a report covering the years from 2001 to 2013, showing that suicide among men has reached its highest levels since the early Eighties, rising sharply in 2007 and hitting a peak in 2013. (In the same period, suicide among women declined, then remained constant.) Always a leading cause of death among the young — for the simple reason that death by natural causes is statistically less likely — suicide rates in men aged between 45 and 59 have also now begun to rise, increasing to their highest levels since 1981.

Former Deputy Prime Minister Nick Clegg earlier this year described suicide as "a massive taboo", the last in mental health. The reasons for this are relatively easy to explain, if not to justify. Before the 1961 Suicide Act, taking your own life was illegal in Britain. It still carries with it a connotation of dishonour and shame, as if those who do it are deserters from the battlefield of life. "The easy way out" they used to call it, and perhaps some still do.

It is still the case that the type of men we think die by suicide are the unwell, the disturbed, the unlucky; who stumble at life's biggest hurdles and are too weak to get back up. Most of us like to think we're made of sterner stuff. We don't know that 75 per cent of people who take their own lives have never been diagnosed with a mental health problem, or that only five per cent of people who do suffer from depression go on to take their own lives.

We think of recent high-profile examples of suicide, like fashion designer Alexander McQueen, who killed himself in 2010, Wales football manager Gary Speed (2011), or film star Robin Williams (2014), as anomalies. Men who had it all: money, fame, the adoration of the public, the respect of their peers, a glorious past and assured futures. We celebrate their memories as professionals, but we don't pause to consider what their deaths tell us about mental health, about the way that no amount of external glory can ameliorate an inner pain.

But the statistics are clear. The problem has become too big for us to ignore. The question to be asked now is not just why young men are killing themselves in record numbers, but what are we going to do about it?

The thing people always want to know is: why?

I am sitting in the almost empty cafe of a no-frills hotel in North London. It's a functional sort of place, designed for people with brief business in the capital rather than those on holiday. A couple of students scan the day's papers. A waitress cuts fruit for the smoothie machine. A man in a suit drifts in and out of the room, talking self-importantly on his phone.

A hand lands on my shoulder. It is Shirley Smith, a small, energetic woman in her late forties with short red hair. I introduce myself and she immediately places my mild accent before pulling me in for a hug. She is from my neck of the woods: Durham, in the North East, where suicide rates among men are the highest in England.

Shirley is a woman I feel I've known all my life. She's like the women who brought me up: warm, tough, unpretentious, prone to conspiratorial laughter. In no time at all we're making fun of the self-important man, cursing the waitress for her noisy fruit blending. And then she tells me about Daniel, her eldest son, who was 19 when he took his life, about how she came home that day in August 2005 with her two other boys — only 10 and five at the time — and how the elder, Matthew, found Daniel with her, how she still can't say the word that describes how he did it, even now, ten years later.

Shirley is in London to visit Westminster due to her role in an all-party parliamentary group for suicide prevention. After Daniel died, she and her family did the only thing they could to try and make some good come of it — they set up a charity, which is now their life's work. If U Care Share — Daniel's brothers and cousin picked the name in the year he hanged himself — works with Durham police to provide practical and emotional support to families bereaved by suicide. It is the only organisation of its kind in England.

"The thing people always want to know is: why?" Shirley explains, her hand slowly turning the cutlery on the table. "But I am yet to meet a family who gets a 'ta da!' moment when you find that out, because the person who can tell them isn't here any more.

"Daniel falls into the group of men who had no diagnosis of mental illness at all before they took their own lives," she continues, "and from the outside looking in, he hid the way he was feeling extremely well. Before it happened I didn't think our son was the type of person who takes his own life. Daniel had all the things you think a person needs: he had a job, he had supportive friends, he had his family, he had plans. But it wasn't enough."

I ask Shirley, as a mother of boys, why she thinks three-quarters of all suicides in the UK are male. She can't say for sure — no one can — but she offers an explanation I will hear in some form from almost everyone I talk to, from scientists and academics to charity workers and families.

"Think back," she says. "We still live in an era where, if a female toddler falls down, her mam picks her up, dusts her down, says 'awww' and gives her a cuddle. A little boy falls down and it's 'you don't cry'. That's what we expect from a man — we want them to 'man up'.

"Daniel learned at a very early age to tie his shoelaces, he learned he needed to be physically strong, physically well. But nobody ever spoke to him — I never spoke to him — about the impact of life."

I ask her hopefully about the government, the people she is here to visit today. Last January, when he made his "taboo" comment, Nick Clegg called for a "new ambition for zero suicides across the NHS". Surely, that is a sign those in power are beginning to take the issue seriously?

Shirley shakes her head. "'Zero suicide' is a fabulous… not 'pipe dream', exactly… I would love for us to truly achieve it. But zero suicide is for people who have contact with mental health services. That's what [Clegg] was talking about — people already diagnosed and in the system. What about the majority of suicide cases, who have no diagnosis at all? They're the people we need to reach. We focus on the tiny percentage in care, because the bigger picture is baffling to people. It's too scary to contemplate."

As we say goodbye, Shirley hands me an If U Care Share pack, which I open later. Among the leaflets, pens and stress balls is a charity wristband, sealed in a plastic case with a card. On it is a photo of a handsome young man with short red hair, a smile playing across his lips, his head cocked slightly as though listening to a joke. Reading the dates under his name, I notice Daniel and I were born just a few months apart.

Men are caught between two very different visions of masculinity - and feel they're fulfilling neither

The reasons behind every individual suicide are unique and complex, but we know common factors and experiences that contribute: relationship breakdowns, bereavement, socioeconomic factors and mental health problems. These factors affect both sexes and have done since the dawn of time, but in almost every country in the world, for as long as records have existed, suicide rates have been significantly higher among men than women, despite the fact that women make a higher number of suicide attempts.

This "gender paradox" is most commonly explained by method. While women are more likely to opt for means such as a drug overdose, which are less immediately lethal, men are more likely to use more violent methods with greater success rates, such as hanging or shooting themselves. Men are also believed to be more naturally impulsive than women, meaning they are more likely to act rashly on suicidal feelings.

More puzzling and difficult to explain is the steep rise in suicide between 2007 and 2013 for British men, while the figure for that period has declined for women. In 1981, men were 1.9 times more likely to die by suicide than women. Today, we are three times as likely. It's an increase charities and academics have been trying to explain since the Office of National Statistics report was published, without arriving at a clear consensus. But theories are emerging. Looking at the wider world in that period, the first and most obvious is the 2008 recession and the subsequent rise in unemployment and poverty. In 2013, research published by the British Medical Journal found English regions with the most job losses since 2008 also saw the largest increase in suicide, while a 2015 Samaritans report found "men from the lowest social class, living in the most deprived areas, are up to ten times more likely to end their lives by suicide." Also this year, academics at Bristol, Manchester and Oxford universities estimated 1,000 extra deaths from suicide and an additional 30–40,000 attempts may have occurred following the economic slump.

Professor Rory O'Connor is the President of the International Academy of Suicide Research and leads the Suicidal Behaviour Research Laboratory at Glasgow University. He has been researching and writing on the topic of suicide for over 20 years and agrees the economy has been a key factor. "It is not that austerity kills, necessarily, but that people who are already vulnerable face another pressure," he says. "If they are worried about their jobs, about not providing for their families, it creates a lot of stress that will impact their well-being."

Anecdotal evidence from suicide helpline staff also suggests a huge rise in callers citing unemployment, job insecurity and benefit cuts as the cause of suicidal thoughts in that period. Women, of course, are victims of economic pressures, too. But research suggests that even today, men attach greater importance to job status and income. A 2014 report by the male suicide prevention charity Calm (Campaign Against Living Miserably), found that 80 per cent of men aged 35–44 consider their job to be "very important" or "important" to their self-esteem, while 42 per cent still believe they should be the main breadwinners in their household (only 13 per cent of women felt the same).

Not only that, but men consider money as central to their appeal to the opposite sex: 40 per cent believed potential sexual or romantic partners are looking for "security/stability", "financial dependability" and/or "hard-working, successful" men. In reality, a survey of women in the same report found "financial stability" ranked far lower (six per cent) than "kindness", "sense of humour" and "trustworthiness".

Which leads to the bigger issue of how men see themselves and their role in the world, and what impact that has on their mental health. One of the most alarming changes since 2007 is that suicide rates among men in the middle age bracket (45–59) have increased by around 40 per cent, meaning they have caught up with their younger counterparts, historically the highest-risk group.

"They are sometimes referred to as a 'buffer generation'," Professor O'Connor explains. "The same group of men who were at risk two decades ago are at risk now, in their middle age. One theory is that it has to do with the way men see themselves in society, and the way that's changed markedly in the last 20 years. The whole idea of what it is to be a successful man is very different now."

A 2012 Samaritans report called "Men, Suicide and Society" suggests this "buffer generation" is "struggling to cope with major social changes". Having grown up believing in the ideal of the silent, strong provider who was probably a war hero to boot — "austere men like their fathers, who believed you don't talk about your problems, you just get on with it," as O'Connor puts it — they now find themselves in a society where such qualities are no longer required or seen as exemplary.

Decades of social progress mean women are more financially independent and less reliant on men, while at the same time, the concept of the "modern man" — open, progressive, in touch with his feelings — has emerged as the new ideal. Caught between two very different visions of masculinity, often struggling financially at a time when they feel they should be at their professional peak, middle-aged men today are also more susceptible to a life event that makes them three times as likely to attempt suicide if it happens: divorce.

The average age of divorce for men in the UK is now 45. The divorce rate began to rocket from the early Eighties onwards and although figures began to slowly decline in 2006, around 120,000 British marriages still fail every year. The Samaritans report states that men in middle age are "dependent primarily on female partners for emotional support", and from the age of 30 onwards tend to have less individual friendships than women. Once they lose their partners, they are less inclined to make new friends or seek help, and become twice as likely to plan to take their own lives.

And while the ONS report showed that the suicide rate among 15–29-year-olds, while still worryingly high, has decreased slightly (the only age group in which that is the case), any hopes that a younger generation are shedding the traditional male tendency to bottle up their feelings may yet prove premature.

An academic study published by the Medical University of Vienna in June 2015 looked at male suicide survivors aged 18–67, and family and friends of suicide survivors. It concluded: "Almost all men reported that their masculine beliefs led to them isolating themselves when they were feeling down, to avoid imposing on others… and instead, relied on coping strategies that required less immediate effort and provided short-term alleviation of problems, for example, drug or alcohol use, gambling and working excessively."

Perhaps most worryingly, "some men reported that adherence to masculine norms meant that feelings associated with being vulnerable provoke greater anxiety than the thought of being dead."

Often it's the loudest voice in the room, the life and soul of the party, that is really struggling

It is a cool, bright sunday morning in July, and along Trafalgar Square, spectators are cheering on exhausted runners as they reach the final stretch of the annual British 10K London Run. Around the corner, the Old Shades pub on Whitehall is filling up with a steady stream of people. Red-faced and limping, smiling and triumphant, they queue patiently for well-earned pints, swapping stories of finishing times and confessing to various aches and pains.

There are 130 of them in total, all decked in the orange bibs of the charity Calm. They have run for sons, brothers, fathers, friends. Many have photos of those they've lost stuck to their backs — men with bashful smiles, captured in moments of joy that proved more precious than they should have.

At the bar I meet David, a big Scot with a ready smile and gentle, friendly manner. He seems, by any external measure, like a man without a care in the world. I automatically assume he is running for a friend or a relative; it seems like the kind of thing he'd do. "No, mate," he says matter-of-factly, "I'm running for myself. I've suffered with anxiety and depression for years."

David tried antidepressants but they didn't work. Then, with some reluctance, he agreed to see a counsellor. "When I first sat down, I told the guy, I have no idea why I'm feeling the way I'm feeling, and no idea what I am supposed to say. But slowly, it works. By talking you join the dots and piece together how you are feeling."

After David leaves, I tell another runner, a woman in her forties, how talking to him surprised me. She nods in agreement. "You hear the word 'suicide' and you immediately imagine a depressed, lonely person. I remind myself, it's often the loudest voice in the room, the outgoing guy, the life and soul of the party, the one who is worrying about whether everyone else is OK, that is really struggling himself."

After an hour or so, Jane Powell, Calm's CEO and founder, gathers everyone together at one end of the pub. It's a tight squeeze so she climbs onto a table to address them. She shouts out that today they've raised £50,000 — a record. A cheer goes up. It's an excuse for another drink.

No other organisation is doing more than Calm to address male suicide in the UK. It stages music and comedy events, distributes a quarterly magazine called Calmzine and runs awareness campaigns that adopt a brisk, no-nonsense tone that has become the charity's hallmark. You may have noticed its "Save the Male" posters, or those comparing the number of British soldiers killed in Afghanistan to men who have taken their own lives (40:50 in 2012), or heard public supporters like rapper Professor Green talking about it on Newsnight (below).

But the main way Calm tackles suicide is by giving men a safe place to talk. Its helpline is manned by eight staff members who work seven nights a week fielding over 5,000 calls and web-chats a month, more than they can handle. Team manager Zoe says the callers range from teenagers confused about their sexuality to lonely elderly men who want to hear a friendly voice before bed.

"Often they say their parents or their friends have given them the 'man up' or 'get a grip' line," she says. "It doesn't mean they don't care, but it does mean they won't talk to them and [they] turn to us instead. It's OK to talk to us because we're strangers. Quite often we never even find out their names." After a couple of hours, the runners in the Old Shades start drifting away. Each one, without exception, finds Jane to thank her, usually accompanied by a hug. According to her, the huge response to Calm's helpline — the only UK one devoted to men — disproves the theory men are incapable of reaching out when they feel distressed. "I think men want to talk, they just want to do it in a different way," she says. "I've been told time and time again, 'men won't ask for help.' And you just think, 'OK, why the fuck are we always struggling to take [all] these calls then?'

Aged 10, Matt came home to find his brother Daniel, his best friend, had killed himself.

Bootham crescent, home of York City Football Club, is an old-school League Two ground in the centre of the city, where you can still stand on the terraces, and adverts for local taxi firms and cleaning companies line the perimeter of the pitch. Upstairs at one end of a narrow function room, a group of players from the under-21 squad are gathering on assorted desks and chairs. They debate their performances and opponents in the last game. One lets out a loud fart and they all laugh. Another walks in dressed in a fresh white tracksuit, and they laugh again. "Didn't know you were coming as Roger Federer!" someone shouts out.

At the other end of the function room, a young guy in a black tracksuit and glasses is setting up a TV screen. Matt Smith, from If U Care Share, assisted his father in running this workshop for three years, before his dad threw him in the deep end one day and told him he was on his own. Since then, he has delivered hundreds of talks at football grounds and schools all over the country.

Matt beckons the lads over. They take their seats — cocky ones at the front, shy ones at the back — four or five rows of spotty faces, gelled fringes and vibrating knees. Matt begins orchestrating a series of small games designed to make the boys think about stress and depression and the way unexpressed emotions can pile up and become overwhelming. Volunteers (the cocky ones), are made to catch balls or wear silly costumes. They watch clips of David Beckham talking about his OCD and the depression that followed his red card against Argentina in the 1998 World Cup. They discuss Gary Speed.

Matt is a natural public speaker. He loves football and the lads sense it. It also helps that, despite being only 21 himself, he somehow appears a lot older. So they more or less respect him, a young guy telling them to talk about their feelings, telling them it's actually OK not to feel OK, as far as their attention spans and their mobile phones will allow.

Then Matt sits down and plays the video. It shows Matt telling the story of how he came home one day, aged 10, to find his brother Daniel, his best friend, had killed himself. The chewing, the texting, the fidgeting all stops. The only sound in the room is a lawnmower buzzing faintly on the pitch outside.

"I used to end this presentation by telling you that suicide is the second biggest killer of young men," Matt stands back up to tell them. "But I cannit. Not any more. Because now it's the number one killer of young men."

Targeting young males involved in sports has emerged as a major tactic in the fight for suicide prevention, with other charities such as Console performing similar workshops at rugby clubs. It is based firstly on the premise that so-called macho men who play sport are less likely to open up in the first place but, more positively, it is to try and harness the support network of a team: the men may not feel suicidal themselves, but they can be taught to spot the signs in those they share a dressing room with.

"It's about trying to bring about a culture change, so that young lads learn to talk about being down or stressed in the same way they would having a bad ankle," says Matt, who believes he should be able to run workshops with boys as young as four.

"There's this fear among men that if we talk about suicide then it might happen, but actually being open and comfortable with the topic could make a big difference."

Charities like Calm and If U Care Share are trying to offer solutions to the male suicide crisis, but as with any area of public health, they can't do it alone. Until the government takes the issue seriously, they will be fighting a losing battle. "The helpline is a bandage and what we do is try and stem the flow," Jane Powell explains. "But the idea that Calm exists for the next 100 years? That's terrible. If we're going to get serious about it we have to change society, otherwise what are we doing?"

Professor O'Connor agrees the government needs to do more. "Speaking as a researcher, suicide is a chronically underfunded area," he says. "Governments have a responsibility not only to make it a public priority on a policy level, but to provide resources and services that actually work."

Overall, while £671m is spent nationwide annually on sexual health initiatives, £160m on encouraging people to stop smoking, and £108m on anti-obesity programmes, only £40m is spent on mental health, within which suicide prevention is just one problem. There is evidence, in other UK regions, of increased funding making real inroads into the issue of suicide. In the past ten years, Scotland has gone from being historically one of the worst affected areas of the UK to the sole region actually bucking the trend. A large part of the reason is that in 2002, the Scottish Government launched "Choose Life", a ten-year national action plan aimed at reducing suicide, which has achieved a 19 per cent fall in incidents.

O'Connor, who helped develop the strategy, explains: "There has been a huge focus [in Scotland] on providing people on the front line of services — police, GPs, psychologists — with applied suicide skills training, so they could identify those at risk.

"At the same time, there was a big anti-stigma campaign, an effort to get the simple message out there that suicide prevention is everyone's business and that talking about it is a good thing. There is this perception among people that goes: 'if I ask somebody if they are suicidal, that will plant the idea in their head.' There is no evidence for that at all. It does quite the opposite."

We are trying to be comic book heroes, rather than real people.

Although there is no simple solution to reducing the number of men in Britain who take their own lives, everyone I speak to agrees there needs to be a concerted effort by the government to learn from Scotland's success and support both gender-specific mental health research and services such as those provided by Calm and the Samaritans. At the start of 2016, pressure is mounting: 200 high-profile figures, including former spin doctor Alastair Campbell and director Danny Boyle, have signed an open letter from Calm calling for equality between physical and mental health treatment ahead of the government's spending review.

But at the same time, the crisis suggests there needs to be a deeper societal shift that we all participate in. We need to re-evaluate our attitude not just towards mental health, but masculinity itself.

I talk to Joseph, one of the runners I met near Trafalgar Square. He's a handsome, articulate 29-year-old with a winning grin.

"I made a suicide attempt when I was 18, and then found myself in a psychiatric ward," he says. "It's still very vivid. I remember two of my best mates visiting, and one of them sitting there in tears. It was one of the most desolate places I've ever been. I don't remember feeling any lower than that."

With the help of his family, Joseph managed to fight his way back. But in order to do so, he says, his perception of what it means to be a man had to change.

"We are trying to be comic book heroes, rather than real people. Still trying to hide behind the football terraces, or the computer game screen. Behind pints, or lifting weights, or talking about how many women we've fucked," he says. "We're not simple creatures, men, but we pretend to be. Because all this time we've been told being a man should be simple. Our dads believed it, and their dads did, and their dads did.

"Sometimes," he continues, "to make fun of women, men say, 'All they do is talk about their feelings.' But we could learn from that. The truth is, our brotherhood is not as safe a place as their sisterhood. I think we need to really look at each other and ask: what are we doing? Are we being ourselves, or what we're expected to be?"

It doesn't sound enough, does it? That the cure for the British male suicide crisis is talking more about our feelings, and ridding ourselves of the unrealistic and outdated expectations we place on ourselves as men. But the truth is that, alongside greater action from our government, these might be the best weapons we have to make sure we lose less men like Daniel.

A few years after Joseph tried to kill himself again, aged 22, his girlfriend gave birth to a baby girl. "When I first knew she was coming, I was terrified," he remembers. "But when she was born, I looked at her and I realised that even if every day was going to be tough as hell, I had to do something about it. I had to make sure I was alive."

For more information: ifucareshare.co.uk; thecalmzone.net; mind.org.uk; consolecounselling.co.uk; samaritans.org

This article was first published in 2015.