Edinburgh’s old Royal Infirmary has been abandoned for more than a decade, its doors boarded up, its gutters overgrown with buddleia and fireweed. Through the scaffolding you can just about make out the way to the old ward 1 and its annexe, ward 1A, where urgent medical admissions were once wheeled on trolleys directly from A&E. It was unusual for a ward to have an annexe, but 1A was set up for a particular clientele: it was the designated ward for anyone who had attempted suicide. I trained in medicine in the hospital throughout the 1990s, worked for a while in the adjacent A&E, and it is not far from the GP practice where I work now.

There was a whiteboard in 1A’s office: the names of the inpatients were listed in a column down one side, then in rows from left to right was written each patient’s date of birth, supervising consultant, any outstanding tests and, last of all, the bleak roll call of the drugs each had taken. Occasionally in place of a drug combination would be written “jumper”, and an estimated height of fall. The beds were arranged in a rough circle; some patients lay with covers pulled over their heads – from a feeling of indignity, perhaps, or to escape the glare of the lights. Others lay hardly aware, the black liquid charcoal they had been obliged to swallow dripping from their mouths, mingling with rivulets of mascara-soaked tears. There was a palpable sense of misery and despair in the small space.

Doctors’ rounds on 1A occurred in stages. First the acute physicians would pass through the night, rigging up intravenous drips of antidote, mitigating the most harmful effects of each patient’s concoction – effects beyond the wildest projections of drug-testing laboratories. At 8am sharp the toxicologists would arrive – physicians with an academic interest in the dramatic effects of exotic drug combinations. The jumpers had a different review: orthopaedic surgeons would turn up after going around their own wards, glance more at the x-rays than the patients and check that plaster of paris was sufficient to restore each to mobility. If not, they would be rolled on trolleys through to the specialist wards to have their bones wired or nailed back into place.

What I have learned from my suicidal patients – podcast Read more

Following the emergency physicians, the toxicologists and the orthopaedic surgeons, just as the hangovers were kicking in, would come the psychiatrists. They would draw curtains around each bed, pull up their chairs and begin to patiently unpick the context of each person’s overdose or their leap from a height. As far as I can remember, neither the physicians nor the orthopaedic surgeons ever enquired as to why suicide had been attempted – that was simply never considered their job. It was the task of the psychiatrists to distinguish those who had attempted suicide on impulse because of some temporary crisis, or under the influence of alcohol or drugs, from those who had serious, ongoing intent to self-harm. As a medical student I would sit in on these consultations. Your feelings are not you, was the subtext; they come and go. Hold on, and the emotion will pass. If your suicidal feelings were deemed fleeting, you would be discharged home; if they weren’t, you would be transferred to the city’s psychiatric hospital. I marvelled at the confidence with which the psychiatrists felt they could differentiate the two groups.

Once a man was brought in who had jumped from the Forth Road Bridge. He had fallen 150ft, shattered his ankles and three of his vertebrae; “A fall on water from a height like that is like falling on concrete,” one of the nurses told me. He had crewcut hair, a scar across his lip and lay as if pinioned to the bed, eyes wide with fear. “We’ll need to keep a close eye on him,” the psychiatrist told me. “No one jumps from a height like that on a whim.” It was not the first time he had tried to end his life; I remember the pinched face of his mother when she came round at visiting time, the drawn bun of her hair, the tremble in her hands as she sat at his bedside.

Later, as a doctor in the adjacent emergency department, it would be my job to break bad news to the families of those who had been rushed to A&E too late or too broken to survive. Horror was a common reaction. Shock, of course, and grief, but so, too, was a kind of wretched acceptance. Often, the bereaved families had previously sat at bedsides on ward 1A, with a brother or mother, sister or spouse, and with the completion of the act there was sadness, of course, but also something akin to, but different from, relief – that a great and unappeasable suffering had finally come to an end.

“Suicide does not necessarily constitute a criticism of the life being ended; it may belong to that life’s destiny,” wrote John Berger in A Fortunate Man, coming to terms with the death of a close friend. “I do not search for what I might have foreseen and didn’t – as if the essential was missing from what passed between us; rather I now begin with his violent death, and, from it, look back with increased tenderness on what he set out to do and what he offered to others, for as long as he could endure.”

I can see the Forth Road Bridge from the north windows of my house: gun-grey twinned towers of 500ft stand more than half a mile apart, suspending four carriageways high over an estuary of ferociously cold tidal water. Figures for how many people jump from the bridge are hard to come by, but have been estimated at as many as 20 a year – comparable to the number of people who jump from San Francisco’s Golden Gate Bridge each year.

A lane for pedestrians and one for cyclists are slung to each side of the bridge; often in the evening I bike over the bridge to Fife to meet my brother. I listen to music as I pedal; each turn of the wheels lifting me higher, out over the water, the landscape broadening so beautifully it can make me feel glad to be alive – the sun setting behind Stirling and the Ochil Hills in the west; deepening lilac light over the North Sea to the east. A couple of years ago, the bulk of the traffic was moved west to a new bridge, the Queensferry Crossing, and so it is usual to have the bridge to myself. From time to time, I have passed solitary walkers glancing over the parapet at the waves below, and wondered whether they are working up the courage to jump. Spaced out along the railings are signs placed by the Samaritans, urging anyone thinking of jumping to give them a call. Whether a plaque can change the mind of someone driven to such an extreme is an open question. We try, say the plaques. We have to keep trying.

In my work as a GP, I regularly hear people tell me they wish they were dead or that they are planning suicide; my task in that clinical moment is to try to make them feel better, of course, but also to put into practice that training from ward 1A, and figure out how seriously they mean what they say. If someone is suspected of being high-risk, they are referred urgently to the local psychiatric team to be offered an appointment within five days. For those who cannot wait that long, there is the Mental Health Assessment Service at the local hospital, staffed 24 hours a day. I have conversations like this two or three times a week, and it is not possible to refer everyone. Depending on how they go, these encounters with suicidal people can make me feel proud of the utility of general practice, or embarrassed by its futility. It feels worthwhile to offer a friendly ear to someone in need, even if it’s just to offload a measure of misery and despair, and good to provide a welcoming, accessible space to voice profound questions about the value and purpose of life. But just as often, I am stopped short by the absurdity of grappling with all the social, personal and historical hinterland of those questions in a 10-minute appointment, squeezed between baby checks and pill reviews, listening to lungs and peering down throats. Perhaps it is hubris to suppose that a general practitioner can ever presume to convince anyone there is value and purpose in life.

In an ideal world, anyone intending to take their life would have prompt access to long consultations with someone trained in psychology or psychiatry, skilled not just in diagnosing mental illness but in choosing antidepressant medication where it might help, and delivering one of the myriad forms of psychotherapy. According to the Samaritans, about 90% of the 5,000-6,000 people who die by suicide each year in the UK are said to be suffering from a mental illness of one kind or another. The remaining 10% presumably choose suicide in full awareness of what they are doing, as a rational choice.

The people who pass through my clinic asking to be cured of their suicidal thoughts are as diverse as our communities: unhappily married spouses, harassed single parents, teenagers, isolated pensioners, adult survivors of childhood abuse. I was taught by the psychiatrists of ward 1A to gently swing these conversations round to explore those social connections that sustain life, then explicitly document them – these get called “protective factors” in the spare language of the clinic. I was to find a way to remind my patient that every suicide punches a ragged hole through our collective social fabric, and that such holes are not easily mended.

Hamlet’s “To be or not to be” was rhetorical in Shakespeare’s day – suicide was forbidden by God and worse than murder. The impulse for self-annihilation was to be deterred rather than treated with compassion: the bodies of suicides were tossed on the town midden, buried at crossroads with stakes through their hearts and, in France under Louis XVI, dragged across town face down through the dust. That suicide implied abject moral degradation was taken for granted when the Scottish philosopher David Hume proposed the contrary view (in his essay On Suicide, published after his death in 1776): that suicide should be permitted, even lauded, as a rational response to unbearable mental suffering: “We may at least be assured that any one who, without apparent reason, has had recourse to it, was cursed with such an incurable depravity or gloominess of temper as must poison all enjoyment, and render him equally miserable as if he had been loaded with the most grievous misfortunes.”

There is a lot hanging on Hume’s use of “incurable”. Antidepressants have flaws, dangers and many side-effects, but there is no doubt that they help some people cope with relentless, distressing feelings of despair and joylessness – particularly when they are used in combination with other kinds of help. Over the 20 years I have been practising medicine, I have seen hundreds of patients recover a sense of worth in their own lives with the help of antidepressants, often in combination with counselling and community support. When I frame suicidal feelings as a passing storm, an illness to be treated, an irrational conviction to be reasoned away, it is buttressed with the memory of those recoveries.

Get the Guardian’s award-winning long reads sent direct to you every Saturday morning

Prompt access to the psychiatric services also helps, and in this, the effects of the last decade’s squeeze on NHS funding is having a noticeable effect. Hume’s essay assumed that “gloominess of temper” was something not amenable to treatment, but we know that the reverse is true – that mental health services get results, and that since 2010 they have faced a double blow: restriction of resources and a blossoming of demand. The cutting of mental health services has had a disproportionate effect on people living in poverty, who are more likely in the first place to suffer mental ill health. A sociological study of the phenomenon concluded last year that “austerity and associated policies have combined to increase the overall burden of mental distress and marginalisation within the UK”. Eight out of 10 NHS trusts have lengthened waiting times to access mental health support since 2016, and for those who get through the door, therapists have been forced to cut both appointment times and access to services.

For a while now I have been asking therapists about suicides among their patients. It helps me in thinking about my own, as I pick endlessly over the antecedents of each, wondering whether there was something I missed (I hesitate to call them the “successful” suicides – the “completed” suicides? the “dead” suicides? There is no easy language). There is often a solemn and melancholic air to that conversation with fellow therapists, a kind of wistful regret mixed with puzzlement, as slowly each begins to talk, to remember, to ask themselves whether anything else might have been done. I have asked GPs, counsellors, psychoanalysts and psychiatrists, impressed but not surprised by the detail with which each story is recounted, even decades after the event. They remember the cast of light of their last conversation with the deceased, their particular facial expression, the tone of voice on the last answerphone message. The suicides I have known have been broadly representative of the phenomenon: mostly men, who have mostly chosen violent deaths, mostly coming from deprived or oppressed communities. About 75% of suicides are male.

I am haunted by the memory of a man brought to me by the police, found teetering on the edge of one of the city’s bridges, the survivor of a harrowing childhood. I managed to get him an urgent appointment at the psychiatric clinic, started him on some antidepressants, saw him as frequently as I could (for a GP, that’s once a week), but he returned each night to a homeless hostel, and one Monday morning I came in to a police report that he was dead. My last meeting with a brokenhearted young call-centre worker, barely able to speak because talking would have involved a pause in the frantic checking of his phone for messages that didn’t come. I made soothing reassurances, offered a prescription to dull his agitation and a review later in the week that he didn’t attend. I followed up by phone, but could not reach him – three weeks later he was dead. The soldier who hiked out on to the moor, lay down in the heather and overdosed. Émile Durkheim, the great sociologist, theorised as to why suicide is so common among military personnel – not simply because of the devastating effects of war, or because of their easy access to weapons, but because of the depersonalising effect of army training: “Military esprit can only be strong if the individual is self-detached, and such detachment necessarily throws the door open to suicide.”

I remember, too, with more gladness, the man who climbed a tree in his garden with the intention of hanging himself, and jumped. The branch broke, and he fell to the Earth as if reborn – those lungfulls of grassy air tasted sweet, he said, and he has never contemplated his own death again. Most had never spoken to me of their plans – Samaritans estimates that only half of UK suicides have ever had any contact with mental health services, and the police report on the desk can be my first intimation of such despair. These episodes are remembered so vividly in part because completed suicides are rare – I can count those I have known in 20 years of practice, on just a couple of hands. But suicide’s seismic aftermath reverberates on through the years; although the acts themselves might be sporadic, the misery they cause may become routine clinical fare, going on through the decades, spreading through families, undermining relationships, setting off aftershocks. A therapist might know only a few clients or patients who have died by suicide, but will have had dozens of patients struggling to come to terms with the suicide of a relative or a friend.

Eric Steel’s The Bridge is a documentary film that examines the phenomenon of suicide from the Golden Gate Bridge in San Francisco. Steel and his team set up rolling cameras, filming night and day throughout 2004 – they caught more than 20 suicides and attempted suicides on film – they are also said to have prevented several. Afterwards, numerous family members of the deceased agreed to be interviewed for the documentary.

Steel’s film has many critics; it is as if he has looked through a journalist’s checklist of “don’ts” when speaking of suicide, and set out to do every one of them. But watching it can be a redemptive experience: family after family confess their anger and frustration over their loss, their irritation and regret, but they also describe their love for the one who jumped, and their gratitude that such suffering has come to an end. One friend berates herself over what she might have done differently, but is followed by another who reminds herself of how hazy the zone between doing little to prevent a friend’s suicide and trying too hard. One interviewee feels he could have had his suicidal friend “locked up or something”, and that might have saved him, but recognises at the same time that to think like that is to invite torment: “I don’t blame myself like that.” Watching the interview, I found myself nodding: if I blamed myself like that for the suicides among my own patients, I don’t think I could go on doing my job.

No one feels happy all the time. In his study, Durkheim concluded “it is wrong to believe that unmixed joy is the normal state of sensibility. Man could not live if he were entirely impervious to sadness.” To feel depressed about the state of your life is to demonstrate capacity to imagine something different, and that spark of imagination can prove a motive to change. Often in the clinic, I find myself reminding my own patients that they are not alone in feeling down or despairing, it just feels that way – a large proportion of those who claim volubly to be enjoying life are faking it. Social media has extended this kind of deception, but behind the closed clinic door you see a different picture of humanity, one broken but with the capacity to mend, one that struggles on against unspoken difficulties, often with great courage.

On 10 May last year, the body of Scott Hutchison was pulled from the waters of Port Edgar, a wartime naval harbour just west of the Forth Road Bridge, now given over to restaurants, rowing clubs and pleasure yachts. Hutchison was the singer-songwriter of the rock band Frightened Rabbit, and for the day he had been missing, his bandmates had been putting out fraught messages of love and support. Hutchison had sung and spoken often about his suicidal feelings, and several of his songs imagined his death. He had last been seen leaving a South Queensferry hotel in the early hours of 9 May, walking in the direction of the bridge. “I’ll float away, down the Forth, into the sea,” goes one of his songs, which concludes: “I think I’ll save suicide for another year.”

Soon after Hutchison’s death, I walked over the bridge in a kind of private tribute, listening to his music, thinking about my own patients who have died by suicide. Up on the bridge, the Samaritans have had some new plaques fitted with their freephone number (116 123): “TALK TO US If things are getting to you – this number is FREE to call round the clock.” There were some new stickers of Hutchison’s face, too, circled with a fragment of a line from one of his songs: “While I’m alive, I’ll make tiny changes to Earth.” Hutchison’s family have launched a charity, Tiny Changes, to raise awareness of this issue.

I felt dwarfed again by the gun-grey towers, the widening sky, the glimmering tidal water beneath my feet. The sounds were of the hum from that new bridge, and the rattle of a commuter train trundling over the rail bridge to the east. “His death has changed the story of his life,” wrote John Berger of his friend. “It has made it more mysterious. Not darker.” Hutchison posted his suicide note online, as a tweet, just: “I’m away now. Thanks.” It is still there on Twitter: a scroll through the responses is enough to see that although social media can deepen isolation and alienation, it can also bring people together. There are testimonials from those who see suicidal feelings as a distortion of reality, who have been through dark periods and found ways back to life, but there is also the contrary view – suggesting, with David Hume, that it can be helpful to think of such desolation as incurable, and be thankful that a great suffering has come to an end. Soon after his death was confirmed, Hutchison’s bandmates tweeted: “There are no words to describe the overwhelming sadness and pain that comes with the death of our beloved Scott but to know he is no longer suffering brings us some comfort.”

When I think of the hundreds of patients I have heard speak of suicide over the past 20 years, whether their own or that of others, and I imagine all those I will no doubt hear in the years of medical practice to come, what seems of most help is not an unwarranted optimism, or a belief that suicide can be right or that it is always wrong, but our flawed human capacity to hold mutually contradictory beliefs and voice them with conviction. When the task in hand is to convince a suicidal patient there is value and purpose in life, then thoughts of suicide are best framed as a shared enemy, a corruption of reality, a manifestation of illness – something to be reasoned away, or quelled with medication. But for the families of the dead, who sit later in the same consulting room, those metaphors of distortion and disease can be unhelpful, even hurtful, and what best replaces them are metaphors of victory and redemption, of suffering followed by release.

Gavin Francis is a GP in Edinburgh, and the author of Adventures in Human Being and Shapeshifters – On Medicine & Human Change, both published by Profile and available at guardianbookshop.co.uk

In the UK and Ireland, Samaritans can be contacted on 116 123 or email jo@samaritans.org or jo@samaritans.ie. In the US, the National Suicide Prevention Lifeline is 1-800-273-8255. In Australia, the crisis support service Lifeline is 13 11 14. Other international helplines can be found at www.befrienders.org.

• Follow the Long Read on Twitter at @gdnlongread, and sign up to the long read weekly email here.