Nothing about Rebecca’s life looks sad. She’s strikingly attractive and professionally successful. I met her in her comfortable split-level flat in Fulham, just after she had started a new job, another rung up the ladder of career and income. Four years ago, when she was 31, a long-term relationship that she had thought would lead to marriage came to a sudden end. She still looks wistfully over her shoulder, but at the same time desperately wants to settle down and have children before it’s too late. “Lots of people can’t understand why I’m lonely,” she says. “I’ve got a good job, a lovely family and lots of close friends. But most of them now are married and taken up with babies. I try to be happy for them, but there’s no one I can ring if I’ve had a bad day; there’s no one for whom I’m the most important one. Things like filling out forms make me feel acutely lonely. Who’s my next of kin? My dad.”

Rebecca has joined the 7m other people in Britain who are trying to find love through the internet. She reckons she’s been on at least 100 dates so far. Every time, she makes an effort – gets “frocked up” as Australians say – but it’s never yet been successful, and she travels home from each assignation feeling “more lonely than if I’d never tried”. Her distaste for the whole business is palpable. Still, faute de mieux, she bashes on.

“How does it feel?” I ask, as she opens her page on the Guardian Soulmates website (which shows that, to date, 1,305 people have viewed her and 356 people liked her).

“It feels pragmatic, and sad. I’m admitting, ‘I’m lonely, and I want to have a family’, and there’s a kind of shame in that.”

She takes me through the profiles of men who have recently joined the site, most with cheeky-chappy nick-names: Curbychup, FoodieGeoff, LieutenantGrey. She shows me how she’s built her own profile, presenting herself as a happy-go-lucky woman who’s well read and widely travelled. “There’s a loneliness in having to present yourself in a certain way, definitely. The distance between the image I give and the reality is getting wider and wider. But if I were to write the truth – that I’m lonely and worried I might not have a family – it would be just the most off-putting thing.”

“So people think of loneliness almost as an infectious disease?”

“Yup. Something like that. Most people find it very, very unattractive.”

“Does anyone on the Guardian site ever admit to loneliness in their profile?”

“ NEVER !”

“Are you sure?”

Rebecca taps the word “lonely” into the search box that allows you to seek out potential partners with particular qualities – Hindi speakers, Old Etonians.

Instantly, it returns the message “No soulmates found”.

According to the Office for National Statistics, Britain is the “loneliness capital of Europe”. For the novelist Deborah Moggach, loneliness is “the last taboo: we talk about everything else, even death, but nobody likes to admit that they’re lonely”. And while loneliness has no physical manifestations, it can be an affliction more harrowing than homelessness, hunger or disease. “The greatest suffering is being lonely, feeling unloved, just having no one,” Mother Teresa wrote. Loneliness is the leprosy of the 21st century, eating away at its victims and repelling those who encounter it.

In Britain 7.7m people live alone. “Thank God London property is so extortionate,” a single, 30-something woman said to me. “I can’t afford to buy alone, so I’m forced to carry on sharing.” The number of baby-boomers – people aged 45 to 64 – living alone is increasing year on year. Seventeen million adults in Britain are unattached. More than 1m older people feel lonely all or most of the time, and most of them do not feel able to admit their loneliness to family and friends. Loneliness is one of the chief reasons people contact the Samaritans, though often callers find it hard to admit it. “People who call us sometimes feel that loneliness is not a good enough reason for calling,” says Nick, a long-term Samaritans volunteer. “They feel ashamed or embarrassed, as though feeling lonely isn’t something serious.” Three out of four GP s say that they see between one and five lonely people a day; only 13% feel equipped to help them, even though loneliness has a detrimental effect on health equivalent to smoking 15 cigarettes a day. Only 22% of us have never felt lonely.

I wanted to discover who these shoals of lonely people were and to get a sense of the texture of their suffering. And I wanted to understand the psychology of loneliness. What does it feel like? Can it be cured? Is it the product of low incomes – or, indeed, of prosperity?

In a street off Portobello Road in London, a battered grey door leads into a hallway adrift with junk mail. Up three flights of stairs, in a book-infested eyrie, the psychologist Adam Phillips – once described as the “Martin Amis of British psychoanalysis” for his razor-sharp intellect and often unsettling work – writes his bestselling books and treats his patients. Most of these, he says, suffer some degree of loneliness and the frantic search for romance may exacerbate the problem. “If one’s living in a culture where a lot of people are lonely,” he says, “there’s going to be a tremendous idealisation of relationships. People are going to want more from each other than they can give. It’s going to produce a compensatory dream of unbelievable ecstatic intimacy. And lots of things can be used to appease this – sex, for example. I think in our culture there’s a lot of sexualisation of loneliness. I think that’s what pornography is, in a sense: a despair about relationship, a despair about real exchange. And loneliness is fundamentally about someone’s belief in the power of exchange: whether we can give each other things that make a difference, whether we can make each other feel better.”

While Phillips does not believe that people are born lonely, or that there is a loneliness “gene”, he is pretty certain that loneliness is very often connected with poor parenting and dysfunctional early relationships: “I think it is very likely”, he says, “that people who are lonely as adults were lonely as children.”

I remember his words when the Samaritans put me in touch by phone with James, an IT entrepreneur and property dealer, now in his mid-40s. Looking back, James explains, he reckons he had begun to distance himself from his parents and their bitterly unhappy marriage when he was about six. By the time they divorced, when he was nine, he was “completely separate” from them: “I was living in the same house as my mother and sister, but I probably wouldn’t spend more than 15 minutes a day in their company. I routinely had meals alone, then went back up to my room and stayed there, alone.” He was solitary at school and university; but it wasn’t until he was in his early 20s, and in his first job, that he realised how completely ill-equipped he was to deal with other human beings: “I didn’t fit in, and I didn’t understand why not. Slowly but surely self-doubt came into play, along with anger and anxiety. It was loneliness in the sense of real deprivation, complete lack of human contact.”

“And what does loneliness feel like?”

“Loneliness is worthlessness. You feel you don’t fit in, that people don’t understand you. You feel terrible about yourself, you feel rejected. Everyone goes to the pub, but they don’t invite you. Why? Because there’s something wrong with you.”

It was when he came to the point of feeling “highly suicidal” that James reached out to the Samaritans, ringing them as often as eight times a day. They helped him to “feel human”, and have been a lifeline to him for over 20 years, including seeing him through a “complete mental breakdown” 13 years ago. He expresses his gratitude to them in substantial financial gifts. Because, for all his awkwardness and isolation, James is a self-made multi-millionaire. Along with Princess Diana, Marilyn Monroe and President Trump – des­cribed by his biographer Tim O’Brien as “one of the loneliest people I know” – he is proof that you can’t buy your way out of loneliness. “However much money you have, you remain constrained by your mental processes,” he says.

It may be that affluence is making things worse. We prize space, privacy and independence, and the richer we get the more of these we can afford, yet their corollary is being alone. Our economy works better if people move around to find work, yet mobility stretches and breaks the bonds of family and community. Phillips told me that “capitalism and a mobile labour market make connections between people very precarious and difficult. In so far as people feel that what they’ve got to do is get on, they are, as it were, encouraged to sacrifice relationship and intimacy.”

But if money can’t shield you from loneliness, poverty can exacerbate it. I met Euan at a soup kitchen in Soho on a chilly evening before Christmas. He used to manage a betting shop, but after a mental breakdown ended up on the streets. “I’m an only child and I’ve always been a loner,” he told me. “To be alone is just what my life is. I feel I don’t deserve to be with people, or to have a relationship.”

“What does loneliness feel like?”

“It’s like being offered a full meal, and not being able to eat it.”

Chris Mahoney is a senior co-ordinator at Home Start, a charity that offers practical and emotional support to families with small children in crisis. “Lots of our mums are terribly lonely,” she says, “especially if they are refugees or asylum-seekers. In fact I would say that probably most of their suffering comes from loneliness.”

At Chris’s office in East Sheen I met Alice and her toddler son, Tom. Alice’s husband works 12-hour shifts as a concierge in a smart block of flats, but his income is low, and Alice has been unable to claim Jobseeker’s Allowance because of mental-health problems. So for several months after Tom was born they were stuck in a tiny studio flat above a restaurant, exposed to carbon-monoxide poisoning. “I couldn’t invite people over,” says Alice. “I thought they’d be thinking, ‘Jeez! How can you let your child live in these conditions?’ At three months Tom hadn’t met another baby, and I was desperately lonely.”

“What does that feel like?”

“It feels like a dark cloud. You don’t want anyone to see you and so you get lonelier: it’s a vicious circle.”

Alice is OK , you might think: at least she has a husband and child. But loneliness in marriage can be bitter. Caroline, now 47 and a successful writer, was married for 12 years to a man who, though never cruel, felt increasingly absent. “He was very gregarious,” she says, “always the life and soul of the party, but really very insecure. When we were alone, he would disappear into himself. He didn’t really either talk or listen. There was nothing I could put my finger on, but in a way that was the trouble: there was nothing.” She remembers sitting on the lawn with him one summer’s day, with their children playing nearby. “I was feeling a little melancholy, and said, ‘it’s the tenth anniversary of my father’s death.’ There was a pause, which I thought perhaps was a sympathetic one; but then he said something about flying to New York the following week, and I realised that, as usual, he just wasn’t listening.”

Caroline’s husband started drinking seriously, and things got worse: “He was never, really, fully, with me. His head was either in the office or full of alcohol. If I hadn’t loved him, maybe it wouldn’t have mattered, but I did, so it was very painful.” Caroline had had a stiff-upper-lip upbringing, and she wanted the marriage to work, so she spoke to no one. “I thought that the more visible the cracks, the likelier it was that the whole thing would crumble. So we went around, for several years, looking like the perfect family, with lovely children and good jobs, but all the time I was feeling so alone.” She put her friendships on ice, because she felt unable to tell the people closest to her how much pain she was in. Then, finally, the marriage broke up, and she was able to talk – “and this awful gulf between me and everybody I cared for closed up, and I wasn’t so lonely any more.”

“What does loneliness feel like?”

“Like being surrounded by a dark void that you have no way of crossing.”

The corroding effects of loneliness become more apparent as we grow older. Literature is awash with lonely spinsters. Take the eponymous central character in Brian Moore’s “The Lonely Passion of Judith Hearne”, who lives in dingy lodgings in Belfast, watched over by paintings of her late aunt and the Sacred Heart. In her early 40s, Judith Hearne is plain, pinched and desperately sad – “a temptation to no man”. She drinks to drown the bitterness of her existence.

“When I wrote Judith Hearne,” Moore told one interviewer, “I was very lonely, writing in a rented caravan, I had almost no friends, I’d given up my beliefs, was earning almost no money and I didn’t see much of a future. So I could identify with a dipsomaniac, isolated spinster.” But that was in 1955. Surely now, 60 years on, we don’t bracket middle-aged single women and loneliness in the same way?

I met Fiona, a sharply intelligent psychotherapist, last autumn, and when I told her what I was working on, she swiftly volunteered that she was “desperately lonely” and would be happy to talk about it. “How old are you?” I asked. “I’m 57,” she replied, “or, as the dating sites would have it, ‘I’m 57 but I feel 27’.” So one quiet afternoon we met for lunch and a walk by the Thames. I sensed that what she had to tell me was going to be painful, so for a good while we talked of anything but loneliness. But eventually, sitting on a bench, I switched on my recorder, and we bit the bullet.

It’s not easy, Fiona conceded, to talk about being lonely: “Mental-health problems and depression are quite fashionable now, but loneliness is not fashionable. There’s something shameful about it – ‘it’s my fault, there’s something wrong with me, I’m a horrible person.’” I mentioned that at a recent dinner in Oxford, a brisk American woman had suggested to me that the solution lay in keeping friendships in good shape: “lonely people need to frexercise.” But Fiona explained that, as loneliness gets a grip, this becomes more and more difficult. “It took me a very long time to actually think of myself as someone who’s lonely,” she reflected, “and I feel I’ve only really done that in the last four years or so. If you have a good social life, and you have people in your life you’ve known a long time, and you make friends easily – which I do – it’s very easy to feel un-lonely because you’re quite busy and you’re not short of interactions with people. But I have found, for whatever reason, that I don’t socialise any more in that way.” It’s partly that friends seem so immersed in their own lives – some are now retiring, moving out of London, becoming grandparents – “so the circle has really narrowed. I just spend an awful lot more time on my own.” And it’s partly that she has come to accept that hectic socialising will never satisfy her deepest longings. “What you really need are people that know you very well, and care about you and are available to you,” she says, “and that you can just contact about anything at any time and I don’t have that, and that’s very lonely. I can’t just pick up the phone and say, ‘Do you want to come over? Do you want to go to the cinema? What are you doing at the weekend?’ That simply doesn’t exist now. I didn’t really notice it happening, but it has. So I’m caught in a vicious circle. If you feel you’re unlovable, you feel you can’t be around people, and this enforces feelings of isolation, and so it goes on.”

Going past childbearing age had brought no relief: “Oh God, it wasn’t a relief to me. It’s an ongoing grief. I thought it would go away after my 30s – I thought, ‘if it doesn’t make biological sense, it won’t make psychological sense’. But in fact it just got worse.”

All she wants now, she says, is to share her life, “in very ordinary ways”, with one other person: “I think the whole meaning of life is sharing and relationships and companionship. It’s almost as if doing things on your own isn’t really doing them. If there’s no one to reflect you or relate to you, it’s almost as if you stop existing.”

“What does your loneliness feel like?” I ask Fiona.

“It feels like a bereavement – like an enormous loss of something. And it also feels suffocating – tight and strangling and suffocating, even though it’s an absence.”

“And what do you do when these feelings become overwhelming?”

“Nothing. I used to make myself go on bike rides and stuff. Now I just try to put up with it. I think, ‘this is it, then. This is what loneliness is’.”

As old age hovers on the horizon, the loneliness strengthens. “I don’t really have anything good to remember,” Fiona says, “I think about not having done any marvellous things, and that’s a sickening thought. I notice tiny things begin to go wrong with me physically – and I think, ‘there’s nobody who cares or knows what I’m doing now. If something bad happened to me, who would know?’”

It is a valid concern. In the autumn last year, the body of 68-year-old Marie Conlon was found in her flat at Larkspur Rise in Belfast. She had been dead for nearly three years. In a statement, her family said they were “shocked and heartbroken” at the death of their “beloved sister”. Call me cruel, but how beloved could she have been if they hadn’t seen or spoken to her since the beginning of 2015? I popped into my local funeral directors to learn how often they were presented with bodies which had lain alone in flats until they began to decompose. The lady in charge that day was wary of my questions, and made me promise not to give her name. But yes, she said, this happens quite regularly – bodies lie undiscovered until neighbours complain of a smell.

If this is shocking, it’s perhaps unsurprising. More than half of men and women over 75 in Britain live alone. Three-quarters of older people say they are lonely, and more than a third of these feel their loneliness is “out of their control”. Most admit that they have never spoken to friends or family about how they feel.

On a fine autumn day, I travel to Rutland to meet 85-year-old Barry, and to have lunch with him in the Finch’s Arms at Hambleton. He often used to come here with his wife, Christine, and though she’s now been dead three years, he still talks of “we” and “our” rather than “I” and “my”. Christine was 15 years younger than Barry, so they always quietly assumed he would go first. Then she developed a brain tumour. “Her sudden death”, Barry says now, “left me in a state of physical shock so deep it defies description. My future became a wasteland full of empty days.”

We live in a society that admires independence but derides isolation. Yet for many old people the two go hand in hand. Back in the summer of 1960, following the death of his wife, Joy, C.S. Lewis wrote of the agony of becoming a free agent. “I’d like to meet,” he wrote to Peter Bide, the priest who had married them, “for I am – Oh God that I were not – very free now. One doesn’t realise in early life that the price of freedom is loneliness. To be happy is to be tied.” This was exactly Barry’s experience. He finds it hard to say where grief ends and loneliness begins, but together he experienced them as “a penetrating hurt that doesn’t dissipate – a mental thing that becomes physical and robs you of all motivation. I got very near to losing the will to live: despair is always knocking on the door for the lonely.”

Other elderly people I spoke to described a similar experience in different ways. For 91-year-old Robbie, living in Kent and a widower since 2012, “loneliness is not having somebody to do nothing with.” He hasn’t been out of his front door, except to hospital, for two years now, and he keeps his television permanently on for company (two-fifths of older people in Britain say that television is their main companion). “A lot of the time, I’m not really watching it. But then something interesting comes on, and I say, ‘Cor, look at that!’, and I turn round, and there’s nobody there…” Vanessa, nearly 80, used to work in fashion. “I still hunt for clothes in charity shops,” she says, “but you can’t hunt for friends.”

“What does loneliness feel like?”

“It freezes you. You can hardly get out of bed. I wake up and think, ‘what the hell shall I do?’ I make little lists, try to tell myself that today is a new day.”

Adam Phillips believes that lonely people exercise a measure of choice: “there’s loneliness, and then there are the uses of loneliness. Loneliness can be a refuge, albeit a miserable one. It can be an avoidance of a lot of things that could feel exciting, but troubling. There can be safety in loneliness.” But if people can face their loneliness head on, there’s the possibility of recovery: “somebody only feels lonely because they’ve had the experience of not feeling lonely. In other words, this is reactive to something – somebody feels lonely because they know they’re missing something they have once experienced. They know there’s something good in the world that might appease their loneliness. That seems to me in and of itself a promising element. So once someone’s feeling lonely, I think, in a way, it’s a sign of hope.”

Sara Maitland, author of the bestselling “A Book of Silence”, and also of a handbook entitled “How to Be Alone”, has lived on her own for 20 years. Her home is in a remote Scottish valley, the nearest shop ten miles away. When she moved here she had never lived alone, and was “eagerly awaiting being thoroughly miserable, and having one more thing to blame my ex-husband for”. Instead, she found herself becoming fascinated by silence, “by what happens to the human spirit, to identity and personality, when the talking stops, when you press the off button, when you venture out into that enormous emptiness.” One thing that happened for Sara was that depression – “that I had assumed, throughout my adult life, was part of my personality” – ceased to trouble her. She now thinks of urban life, and being surrounded by people, with horror.

Sara’s isolation – with black-faced sheep for neighbours and no mobile-phone signal – is extreme. But I kept wondering, after speaking to her, whether there were less radical ways in which people who are alone can learn to convert the desolation of loneliness into the richness of solitude. I met Laurence Freeman, a Benedictine monk and an internationally renowned teacher of meditation, in a quiet square in Islington, and was quickly struck both by his profound understanding of human nature and the distinction he made between solitude and loneliness. For him, loneliness is a “failed solitude”. In his experience loneliness contains a “terrible feeling of failure, and there’s shame in that. Lonely people feel they should be connected, and if they feel disconnected, alienated, then that must mean they’ve made a mistake – or that they’ve been pushed into this by fate, or by something they’ve done. This can often involve a combination of paranoia and a very high level of judgmentalism about others. So they’re trapped both ways: they feel judged and they’re also judgmental.” Solitude, he believes, is “the discovery and acceptance of your uniqueness. It’s not just having knowledge about yourself, it’s actual experience of being, which is what we taste in meditation.” This need not be rooted in religion – his meditation techniques are taught and practised in schools and prisons up and down the country. “Solitude in this sense is the basis of relationship – entering into the solitude of oneself, one’s uniqueness, prepares you for deeper, more authentic relationships.”

“So can you learn not to be lonely?”

“I think you can – by embracing solitude. But I think that coming out of loneliness is hard work.”