On a humid July evening, a young woman in a red dress made her way along the banks of the Seine, walked beneath the Pont Neuf and, tucking auburn tresses behind her ears, sat down next to me. She held her shorthaired terrier under one arm and Susan Sontag’s The Benefactor (1963) under the other. She introduced herself in a hybrid English accent typical of international boarding-school students who think of home not as a singular place but as something seasonal – London in the autumn, the Austrian Alps in winter. Then she introduced her pup. ‘His name is Fortuné,’ she said, extending his paw for me to shake.

I have long thought of myself as a lonesome person, but only that summer, when I met Joséphine, did I begin to understand the true depths of human loneliness. Joséphine had come to Paris three months prior, after reading economics at Cambridge; I had come from Oxford where I read history and thus we hit it off quickly the way that only foreigners who meet in a foreign land are able to do. She lamented that, so far, she had spent each night in her own company, sitting on the terrasse at the Café de Flore, taking the same Niçoise salad and the same Pinot Grigio, and watching couples and groups of friends pass by.

In the weeks following, Joséphine phoned me to join her for dinner, to see her apartment’s library or to attend a variety of events that seemed of dubious legitimacy (a masked ball at her apartment, a boat race outside Paris, a dinner at her estate in Bavaria) and, while I continued to meet with her twice a week on the same bench at sunset, not once did I join her elsewhere.

I can’t say why I declined her invitations: I had come to Paris for solitude and was fearful of allowing another to breach it, but it didn’t end up mattering. As I’d suspected, there was no elsewhere – no boat races, no estate. Joséphine had wanted only someone with whom to talk. She admitted this one evening, as the summer came to a close. Then she stopped showing up. She stopped phoning, too.

Loneliness is a relatively new concept in academia, beginning to trend in the mid-1960s, and becoming prominent only with Robert Weiss’s all-important Loneliness: The Experience of Emotional and Social Isolation (1973). But loneliness studies did not commence in a uniform, rigorous way until 1978, when the creation of a 20-item scale to measure one’s subjective feelings of loneliness and social isolation – the so-called University of California, Los Angeles Loneliness Scale – lent accuracy and comparability between publications.

Still, loneliness remains a slippery concept. After God creates Adam, God says: ‘It is not good that the man should be alone: I will make him a help mate.’ Perhaps with the universality of Genesis in mind, the philosopher Ben Lazare Mijuskovic writes in Loneliness in Philosophy, Psychology, and Literature (2012): ‘[M]an has always and everywhere suffered from feelings of acute loneliness.’ Yet loneliness means different things to different people. Some people feel lonely spending one night alone; others go months with minimal communication and don’t feel a thing. ‘Some may be socially isolated but content with minimal social contact or actually prefer to be alone,’ writes Julianne Holt-Lunstad, the lead author of a 2015 report on loneliness in Perspectives on Psychological Science. ‘Others may have frequent social contact but still feel lonely.’

In spite of such variation, most people don’t choose extended loneliness, or lengthy periods of uninvited solitude, and to hear this unwanted state romanticised – called ‘beautiful’ – is a particular type of sting, the way someone who has been fired or recently divorced might wince at being told it is ‘for the best’. Indeed, there are many serious drawbacks to longterm loneliness, from severe depression to irreparable cognitive damage. In a study on the subject, Holt-Lunstad aggregated data from a range of independent studies within which participants were followed for an average of seven years. She found that people who were socially isolated, lonely or living alone had a roughly 30 per cent greater chance of dying during the study period than those who had ‘regular social contract’.

the very technologies that promise to connect us to others serve to quarantine us from opportunities to make genuine connections

Interestingly, much of the idealisation of loneliness in art and literature turns out to be a façade. Henry David Thoreau rhapsodised his alone time. ‘I find it wholesome to be alone the greater part of the time,’ he wrote in Walden: Or, Life in the Woods (1854). ‘Why should I feel lonely?… I am no more lonely than the loon in the pond that laughs so loud, or than Walden Pond itself.’ Lo! How romantic to be alone! he begs his reader to think. And yet, Walden Pond sat within a large park that was often swarmed with picnickers and swimmers, skaters and ice fishers. In his ‘isolation’, Thoreau corresponded frequently with Ralph Waldo Emerson; and he went home as often as once a week to dine with friends or eat the cookies his mother baked. Of course he wasn’t lonely: he was so seldom truly alone.

Still, it’s unfair to blame Thoreau – or anyone else who flirts with loneliness yet falls short of truly engaging with it. Loneliness can be a miserable state and people, accordingly, work hard to avoid it. Over the past three decades, Americans have reported decreasing levels of loneliness, and one can assume that this holds true for other first-world countries, where a stream of invention works both directly and indirectly to prevent it: social media, artificial intelligence, virtual reality. The promise is that one can always be connected, or more accurately, constantly engaged in the simulacrum of companionship as mediated by iPhones, the internet or, sometime soon, an artificial being. But, as Olivia Laing shows in The Lonely City (2016), the very technologies that promise to connect us to others serve to sever us, even quarantine us, from opportunities to make genuine connections.

Loneliness can be hell – why would we want any part of it?

Yet there is a central paradox around loneliness. While it can lead towards very undesirable places (isolation, depression, suicide), it can also make us better observers of the social world. We can become more perceptive, more in charge of our own reality, as loneliness makes life compelling. Vitally, loneliness assures us that our life is our own. Historically – and mythically – it has been the singular and narrow path towards virtue, morality and self-understanding.

In The Epic of Gilgamesh, written circa 2100 BCE, it is only once the gods kill Enkidu that his friend and fellow traveller Gilgamesh can complete the journey to find the source of eternal life. And only when Christ spent 40 days and 40 nights alone in the desert, confronting the devil – without aid of God or angels – was he able to prove that he could resist all temptation. The ‘angels came and attended him’, says the scribe who wrote Matthew, but only once Christ had succeeded in his task.

Unable to escape the throngs of people querying him for advice and prayers, the 5th-century sage Saint Simeon Stylites, spent 37 years sitting atop a pillar on a one-square-metre platform outside Telanissus (modern-day Taladah, Syria). For sustenance, Simeon relied on young boys from town to climb up the pillar and pass him small parcels of bread, water and goat’s milk. He believed that if he could not escape the busyness of the world on the ground, perhaps he could be alone – and truly in control of his world and his thoughts – if he were closer to the sky. (He eventually built the pillar nearly 50 feet high).

Not everyone safely returns from loneliness, but those who do – those who retreat into themselves and successfully reemerge into society – return with a far greater understanding of themselves and of others. In loneliness there is thus a balance to be struck; it poses at once the highest risk and the highest reward.

Of course, experiencing loneliness does not uniformly create virtuous and moral humans, but it has a variety of other benefits: according to a study published in 2015, feeling socially isolated (or on the social perimeter, as any outsider artist or writer might attest) leads to increased attention and surveillance of the social world and a heightened ability for observation.

Using electrical neuroimaging with a small test group, Stephanie Cacioppo, assistant professor of psychiatry at the University of Chicago and her husband John, also at Chicago, found that participants who described themselves as particularly lonely reacted to pictures of threatening stimuli more than twice as fast as people who described themselves as non-lonely (in the lonely: about 116 milliseconds after stimulus onset; in the non-lonely: about 252 milliseconds after stimulus onset). As John Cacioppo writes in a similar study, this suggests that lonely people’s ‘attention is drawn more to the distress of others’.

The fact that lonely people are more attentive to other people’s distress – principally on a subconscious level, as evidenced by the speed at which the reactions took place – implies that lonely people have a greater capacity for empathy. Ironically, it might be as a result of being lonely that one can better understand others and their social world.

The best creative minds and most charismatic people also tend to be rather lonely. Sharon H Kim, an assistant professor at Johns Hopkins University in Baltimore who focuses on individual and group creativity, recently found evidence that people tend to be more creative if they have been socially rejected. What is perhaps most interesting about Kim’s findings is that no actual social rejection has to have taken place; the creative need only feel rejected in some way. Creativity, Kim claims, stems from the ability to make unique connections, to bind together disparate information. Yet it is predominately the rejected and lonely who are able to best accomplish this. ‘Creative people are better at recognising relationships, making associations and connections, and seeing things in an original way – seeing things that others cannot see,’ writes the neuroscientist Nancy C Andreasen at the University of Iowa in an article for The Atlantic. Often, she notes, the sole way to access creativity, charisma and new ways of thinking is to experience loneliness.

We maintain various states of loneliness: giving up a sense of home, creating only temporary friendships, having meaningless sex. These actions might seem negative but they are related to self-preservation

In evolutionary terms, experiencing loneliness is a part of being human. Lonely, but not too lonely – a Goldilocks mixture that allows us to be at once our individual selves and a functioning part of a wider social sphere – is a key part of survival, according to a recent study from Pamela Qualter, a reader in developmental psychology at the University of Central Lancashire.

Qualter’s findings suggest that, after a bout of loneliness, a ‘reaffiliation motive’ kicks in, which biologically compels one to reconnect with others, once one crosses a certain, individually varying threshold of loneliness. This compulsion holds across all ages, and tends to render loneliness a fleeting experience. Without this motivation to reconnect, we risk being left in a lurch – lonely with no desire to escape it; and yet, without these painful but vital feelings of loneliness, every so often a fundamental part of being human is lost.

‘The process of loneliness helps make us who we are as a species,’ John Cacioppo told me by phone. Without loneliness, we tend to think only of ourselves and not desire the same kinds of connections with others. He added: ‘People who are unable to be lonely are at the highest risk of being psychopaths.’

There are many ways in which humans maintain – intentionally and unintentionally – states of loneliness: giving up a sense of home, creating only temporary friendships, having meaningless sex. While these actions might seem negative on the surface, they are decisions that are unconsciously related to self-preservation. The self dissolves when it is spread too thin, when it is obliged to deal with the glut of acquaintances and jobs, and all the places where one might not be alone but in which one might still feel lonely.

Seeking isolation, searching out the existential pain of loneliness, writes Mijuskovic, is ‘a defensive device to thwart the threat of diffusion, of the self’s evaporation before the overwhelming presence of the “others” as it is assaulted by an impersonal, bureaucratic, industrialised, mechanised society or by violent and traumatic interpersonal relations’.

Although it is distressing to ponder, what happens if everything that comprises a personage – all that one loves, hates, desires, hopes for – becomes only the distillation of other people’s feelings? What if one becomes only a weak prism, reflecting the light of those who have risked diving deeper into themselves? What happens if we do not risk loneliness ourselves? The loss of identity, surely, is a more troubling prospect than loneliness with its risks and pain and drawbacks. For who are we if we cease to be ourselves?

I often think about loneliness, how it can be devastating, but also how it can be a space of reflection that is hard-won; a form of wisdom, a master emotion that colours all other emotions. Importantly, I now feel that without a willingness to face loneliness we forfeit our freedom.

At my loneliest, I have strolled late at night through the less august parts of town, near Belleville and the Père Lachaise Cemetery in Paris, turning every fibre of my being inwards so that I can feel everything, and I’ve discovered an almost infinite hope for the life outside myself. The more I shrink into myself, the larger and more possible the universe becomes.

There is an artwork by Cy Twombly called Untitled (1970) that he made by sitting on the shoulders of his assistant and long-time companion, Nicola del Roscio, who shuttled back and forth along the length of the canvas, allowing Twombly to create four fluid rows of continuous lines in wax crayon. The lines are, Twombly once told the critic David Sylvester, ‘the feeling, from a soft thing, a dreamy thing, to something hard, something arid, something lonely, something ending, something beginning. It’s like I’m experiencing something frightening, I’m experiencing the thing and I have to be at that state because I’m also going. I don’t know how to handle it.’

This has always struck me as the keenest depiction of loneliness’s precarious balance – a movement from what is soft and easy into the realm of the frightening and seemingly infinite; yet, when one stands back from it, one realises that the canvas could not possibly be compelling without all of it.

Not being lonely is a comforting feeling. But I knew that something sacred was leaving me

When I moved from Paris to New York, I came for a job that seemed prestigious and for the chance to enter ‘the real world’ as my father likes to call it, but I also came to do away with my loneliness, to be in a skyscraper with Anglophone co-workers and meet friends for drinks in the evening, as normal people do. I became a good deal less lonely, and yet I also became emptier – emptied of the time for reflection, that continual reminder of my capacity for emotion that I’d hoped to parlay into some form of happiness.

I tried to find pockets of loneliness. I went for strolls, up and down Manhattan, but even alone on the streets, streaked with bustling humans who didn’t turn a thought towards me, I couldn’t feel the same kind of loneliness as I had in France. There were simply too many people to talk to – too many text messages, and old friends coming to visit, and parties to be dragged to. I felt my freedom being closed down, my mind’s ability to wander, to make disparate connections, being moulded. It felt good. No question. Not being lonely is a comforting feeling. But I knew that something sacred was leaving me.

Joséphine called recently, from London, where she had enrolled for Master’s at the Royal College of Art. The conversation was short. She said she was studying in a café inside Somerset House. She told me that not once had she spoken to someone who was not a server or a clerk or a taxi driver for the rest of her time in Paris that summer. She had made no more attempts to commune with others; Fortuné was still her main source of company.

Back at university now, she felt loneliness closing down on her like ‘the white on the lid of a box’ – a version of a quote I later found in a short story by F Scott Fitzgerald. And yet, she said she was ‘more aware’ as she put it – better at understanding herself and her world. None of her social jockeying, her claims to extravagance, could get her as far as spending time with herself, even when it proved most painful.

Then, she quoted Sontag to me – whom she had been reading on that first day along the Seine. ‘Alone, alone,’ she said, ‘I am alone – I ache … Yet for the first time, despite all the anguish and the reality problems, I’m here. I feel tranquil, whole, ADULT.’

Loneliness is hell; I knew that. And yet, I couldn’t help but crave it once more. If only for a moment.