Human societies, at all times and places, have organised themselves around the will to live with others, not alone. But not any more. During the past half-century, our species has embarked on a remarkable social experiment. For the first time in human history, great numbers of people – at all ages, in all places, of every political persuasion – have begun settling down as singletons. Until the second half of the last century, most of us married young and parted only at death. If death came early, we remarried quickly; if late, we moved in with family, or they with us. Now we marry later. We divorce, and stay single for years or decades. We survive our spouses, and do everything we can to avoid moving in with others – including our children. We cycle in and out of different living arrangements: alone, together, together, alone.

Numbers never tell the whole story, but in this case the statistics are startling. According to the market research firm Euromonitor International, the number of people living alone globally is skyrocketing, rising from about 153 million in 1996 to 277 million in 2011 – an increase of around 80% in 15 years. In the UK, 34% of households have one person living in them and in the US it's 27%.

Contemporary solo dwellers in the US are primarily women: about 18 million, compared with 14 million men. The majority, more than 16 million, are middle-aged adults between the ages of 35 and 64. The elderly account for about 11 million of the total. Young adults between 18 and 34 number more than 5 million, compared with 500,000 in 1950, making them the fastest-growing segment of the solo-dwelling population. Unlike their predecessors, people who live alone today cluster together in metropolitan areas.

Sweden has more solo dwellers than anywhere else in the world, with 47% of households having one resident; followed by Norway at 40%. In Scandinavian countries their welfare states protect most citizens from the more difficult aspects of living alone. In Japan, where social life has historically been organised around the family, about 30% of all households have a single dweller, and the rate is far higher in urban areas. The Netherlands and Germany share a greater proportion of one-person households than the UK. And the nations with the fastest growth in one-person households? China, India and Brazil.

But despite the worldwide prevalence, living alone isn't really discussed, or understood. We aspire to get our own places as young adults, but fret about whether it's all right to stay that way, even if we enjoy it. We worry about friends and family members who haven't found the right match, even if they insist that they're OK on their own. We struggle to support elderly parents and grandparents who find themselves living alone after losing a spouse, but we are puzzled if they tell us they prefer to remain alone.

In all of these situations, living alone is something that each person, or family, experiences as the most private of matters, when in fact it is an increasingly common condition.

When there is a public debate about the rise of living alone, commentators present it as a sign of fragmentation. In fact, the reality of this great social experiment is far more interesting – and far less isolating – than these conversations would have us believe. The rise of living alone has been a transformative social experience. It changes the way we understand ourselves and our most intimate relationships. It shapes the way we build our cities and develop our economies.

So what is driving it? The wealth generated by economic development and the social security provided by modern welfare states have enabled the spike. One reason that more people live alone than ever before is that they can afford to. Yet there are a great many things that we can afford to do but choose not to, which means the economic explanation is just one piece of the puzzle.

In addition to economic prosperity, the rise stems from the cultural change that Émile Durkheim, a founding figure in sociology in the late 19th century, called the cult of the individual. According to Durkheim, this cult grew out of the transition from traditional rural communities to modern industrial cities. Now the cult of the individual has intensified far beyond what Durkheim envisioned. Not long ago, someone who was dissatisfied with their spouse and wanted a divorce had to justify that decision. Today if someone is not fulfilled by their marriage, they have to justify staying in it, because there is cultural pressure to be good to one's self.

Another driving force is the communications revolution, which has allowed people to experience the pleasures of social life even when they're living alone. And people are living longer than ever before – or, more specifically, because women often outlive their spouses by decades, rather than years – and so ageing alone has become an increasingly common experience.

Although each person who develops the capacity to live alone finds it an intensely personal experience, my research suggests that some elements are widely shared. Today, young solitaires actively reframe living alone as a mark of distinction and success. They use it as a way to invest time in their personal and professional growth. Such investments in the self are necessary, they say, because contemporary families are fragile, as are most jobs, and in the end each of us must be able to depend on ourselves. On the one hand, strengthening the self means undertaking solitary projects and learning to enjoy one's own company. But on the other it means making great efforts to be social: building up a strong network of friends and work contacts.

Living alone and being alone are hardly the same, yet the two are routinely conflated. In fact, there's little evidence that the rise of living alone is responsible for making us lonely. Research shows that it's the quality, not the quantity of social interactions that best predicts loneliness. What matters is not whether we live alone, but whether we feel alone. There's ample support for this conclusion outside the laboratory. As divorced or separated people often say, there's nothing lonelier than living with the wrong person.

There is also good evidence that people who never marry are no less content than those who do. According to research, they are significantly happier and less lonely than people who are widowed or divorced.

In theory, the rise of living alone could lead to any number of outcomes, from the decline of community to a more socially active citizenry, from rampant isolation to a more robust public life. I began my exploration of singleton societies with an eye for their most dangerous and disturbing features, including selfishness, loneliness and the horrors of getting sick or dying alone. I found some measure of all of these things. On balance, however, I came away convinced that the problems related to living alone should not define the condition, because the great majority of those who go solo have a more rich and varied experience.

Sometimes they feel lonely, anxious and uncertain about whether they would be happier in another arrangement. But so do those who are married or live with others. The rise of living alone has produced significant social benefits, too. Young and middle-aged solos have helped to revitalise cities, because they are more likely to spend money, socialise and participate in public life.

Despite fears that living alone may be environmentally unsustainable, solos tend to live in apartments rather than in big houses, and in relatively green cities rather than in car-dependent suburbs. There's good reason to believe that people who live alone in cities consume less energy than if they coupled up and decamped to pursue a single-family home.

Ultimately, it's too early to say how any particular society will respond to either the problems or the opportunities generated by this extraordinary social transformation. After all, our experiment with living alone is still in its earliest stages, and we are just beginning to understand how it affects our own lives, as well as those of our families, communities and cities.

• Going Solo: The Extraordinary Rise And Surprising Appeal Of Living Alone, by Eric Kinenberg, is published by Penguin Press at £21.

Colm Toibin, 56

Colm Tóibín: 'No one told me that I would be most happy in my life when I modelled myself on a nun who runs her own cloister and is alone in it.' Photograph: Eamonn McCabe

No one told me when I was small that I could live like this. No one told me that by the age of 56 I would know all of the gay bars in New York city, most of the Irish ones and a good number of other bars, such as they are, in between. And that I would be content on a Friday and Saturday night at around 10 o'clock merely to feel that those bars were all still there, still full of people calling for more, while all I wanted was to be alone in bed with a book.

No one ever told me that I would be most happy in my life when I modelled myself on a nun who runs her own cloister and is alone in it, not bothered by the chatter of other nuns, or by the demands of reverend mother.

On Saturday I wake at six and relishing the day ahead. I teach on Mondays and Tuesdays; I have to reread a novel for each class and take notes on it. Nothing makes me happier than the thought of this. I often lie there until the seven o'clock news comes on, grinning at the thought of the day ahead.

All day I will read and take notes. The worst-case scenario is that I might need another book, and this involves lot of decision-making and self-consultation. It might end in a five-minute walk to the university library. But normally I go nowhere except to the fridge if I am hungry to see what's there, or to the sofa to lie down if my back is tired, or to the rocking chair if I feel a need to rock.

Normally there's not much in the fridge. In the kitchen there is an oven I have never opened. And there are pots and pans whose purpose may be decorative for all I know. But I know where all my notebooks are. They are all over the apartment. That is the best part. I can leave them where I like and no one touches them or wants to put them away anywhere. No one sighs about books and notebooks piled up. All of the notebooks have stories half-written in them, or stray sentences in search of a home, or musings that are none of anyone's business. If I like, I can go to one of them and add some paragraphs. I don't have to excuse myself, explain myself, or put on a distracted writer's look in order to get down to work. Or worry that someone has, in my absence, opened one of my notebooks and found that they don't like the tone of what is written there.

No one told me when I was small that there would come a time in my life where people would be judged by the quantity and quality of take-out menus for local restaurants. And that I could, without consulting anyone, at any time, make a phone call, order some food, and it would soon arrive at my door.

And then there is music when night falls. I can put on whatever I like, follow dark obsessions without worrying about depressing anyone else, or cheering them up for that matter. There is no one to question my sanity, my taste in music, or say: "That again? Not that again. Did we not hear that yesterday?"

And then there is the small question of alcohol. No one told me when I was a teenager that there would come a time when I would not bother drinking. No one told me that when Saturday night came, I would long to talk to no one and wish to go to bed early, and that my only moment of pure and capricious pleasure would be taking a book to bed that was not for class the next week. Otherwise, my life as a nun is a lesson to others, a pure example of good example. It has its rewards in the morning when I wake in silence with a clear head, ready for more.

Colm Tóibín is an author.

Carmen Callil, 73

Carmen Callil: 'Living alone means freedom, never being bored, going to bed at eight if I feel like it.' Photograph: Felix Clay

I have never given much thought to living alone, because it wasn't something I decided upon, it happened to me naturally. What with a childhood amid a vast family, then the convent, I was rarely alone. I shared a bedroom with my sister, life with my brothers and mother. One set of grandparents lived next door, the others across the road. Many aunts, uncles and cousins were only a yell away. The convent was black with nuns, its dormitories and classrooms packed with other girls. I left home when I was 21.

Almost immediately, I fell in love with a man who was, vaguely, married. An open marriage, it would be called today. For a decade or so, I wanted to be available for him, so I moved into a bedsit above a salt beef bar in St John's Wood. That was 1964. I was 26, and I have lived alone since.

I very much liked being in love and repeated it all too frequently. But I also hated it. I have a photograph of myself aged two, in a pram outside Melbourne zoo. My chubby legs are battling to get out: the look of struggle on my baby face is tremendous. That is how I felt each time I fell in love and spent extended periods with the beloved object. Often it was boredom: hours spent doing what the beloved object wanted, rather than pursuing the thousand things juggling in my own head. When I was in love and thought of marriage, I always came to feel like that child in the pram.

Tussling with this incapacity came to an abrupt end once I started to work. I had been raised to think of work as a prelude to husband, children, home. Once I started Virago, in 1972, and then, from 1982, working at Chatto, too, boredom vanished, and the days and years fled by.

What do I like about living alone? The greatest blessing is the number of friendships you can indulge in, the number of people you can love. I love to hear their stories, follow their lives. This can become frenetic but you can always cross through a night in the diary with BED in capital letters and there is no one to say nay to that. I wouldn't have minded having the children I could have had, but I have insufficient self-esteem to need any duplication of myself in the world. In truth, I have fretted more about my friends, my work and about understanding what is going on in the world than I ever have about failing to "wax fat and multiply", as the Catholic marriage service instructs.

Living alone means freedom, never being bored, going to bed at eight if I feel like it, feeding myself as I like, thinking, pottering and yelling at the radio without feeling a fool. I am never lonely as long as I am at home. I can decorate my house to suit my eccentricities – not everyone wants to live with 200 jugs and thousands of books. Every object in my home reminds me of one loved person or another. Knowing all my friends are dotted around, going about their business but available at the end of a phone is enough.

There are, and have been, great tediums. Men – Auberon Waugh and Lord Longford spring to mind – have occasionally insisted to my face that I was lesbian. I felt this to be an insult to women who are lesbians as well as to myself. I hate getting invitations addressed to "Carmen Callil & Friend" and am often tempted to bring my dog.

But there is so much to do, and to think about, and so many friends to love. They are my rock. If I am in trouble, they help me, and I don't – and never have – worried about dying alone, because everyone does.

Carmen Callil is a publisher and author, and founder of Virago Press.

Alex Zane, 33

Alex Zane: 'It's not about selfishness, just knowing what you like and doing what you want without having to take another person into account.' Photograph: Rex

Having lived alone for the past six years, sharing my home with anything bigger than a cat is not something I enjoy.

This doesn't make me an oddball. I'm not Norman Bates, wandering around my flat dressed as my mother – I just like the fact that if I wanted to, I could.

Living alone provides me with the time I need to recharge, and to let loose the aspects of my personality best labelled "Not For Public Consumption". When Superman needs a break from saving the planet, some time to himself, where does he go? His Fortress of Solitude in the Arctic Circle. I have what I like to call my Flat of Solitude in north London. I'm not comparing my average day to the conquests of the last son of Krypton, but he has a public image to keep up, and that I can relate to.

"Me" is the very best part of living alone. It's not about selfishness, just knowing what you like and doing what you want without having to take another person into account. OK, that sounds selfish, but if you're going to be selfish, it's probably best to do it on your own, so no one knows.

My solitude is not total. I have a girlfriend, and we've been together for a length of time that makes people wonder why we don't share a home. The truth is, she stays with me often. She has a drawer. She knows where I keep the sugar. I know to put the toilet seat down. She knows which of the three remotes actually turns on the TV. I know she checks my internet history.

It's a well-oiled machine. And although it has yet to be spoken out loud, I'm aware eventually a change will come. A change that will involve me no longer eating packets of microwavable rice and soy sauce for every meal. The spectre of co-habitation is looming on the horizon.

There are, of course, some things that I won't miss about solo living. There are moments of melancholy, the silence can be quite over-powering, and if I've spent three days holed up in my flat, when I finally emerge the first conversation I have with another human can be an awkward affair, like learning to speak all over again: "I… OK… you, yourself, well?"

But there's one thing that dwarfs all the other downsides to living by myself, one thing I'll be happy to leave behind. It's to do with my Wii. I try to shake the feeling, but I can't. Ultimately, there is no more tragic image than a man standing in the middle of his living room, alone, in his boxer shorts, pretending to ski jump.

Alex Zane is a DJ and television presenter.

Esther Rantzen, 71

Esther Rantzen: 'Although I'm getting used to living on my own, I still think it's not natural.' Photograph: Karen Robinson

I am living alone for the first time at the age of 71. Until now, most of the changes that arrived with age were mercifully gradual – the need to turn the television volume a bit higher, say, and the first few grey hairs – but this change has been huge, sudden and, for me, cataclysmic.

All my life I have been surrounded by people. As a child, I grew up in an extended family. At college, I lived and worked in a lively and energetic community. Moving into a flat with a flatmate, starting a family, having a bath or going to bed at night, I had company and conversation. Now, for the first time, I come home to an empty, silent flat, nobody to shout a cheerful hello to, no one to listen to the stories of my day. It's been nine months on my own and a difficult adjustment. But I'm getting there.

My life has followed a pattern familiar to most of us as we grow older. You lose a partner; in my case my beloved husband Desmond Wilcox died. Children leave home and create their own lives; my older daughter, Emily is taking a mature student's degree; Joshua, the doctor, works in the West Country; Rebecca, the TV reporter, lives with her husband and they are expecting their first baby.

I mustn't nag them to spend more time with me. So instead I have found ways of making aloneness feel less lonely. Downsizing from my family home to a flat was a help. Not only are there no more empty bedrooms, but given far less space, the pictures and ornaments that mean the most to me are always in my eyeline. The print my mother gave me is on my bedroom wall, instead of downstairs in my old study, so it greets me as soon as I wake. The vase my best friend gave me is on my table instead of being stashed away in a cupboard.

Getting to sleep by yourself is a problem, but I decided not to have a bedroom television. I tried it for a while and although Newsnight was the perfect cure for insomnia, I loathed waking up at dawn with the screen blaring at me. So I fall asleep to Classic radio, which accompanies my dreams with decent music.

I understand why an American survey of more than 300,000 old people found that loneliness is as bad for your health as smoking. You may have spent a lifetime looking after your family; now that they don't need you, it seems pointless to look after yourself. Cooking for one seems too much effort – I can't muster the energy or enthusiasm to make hot food for myself. Cheese and biscuits and fruit fill the gaps.

Although I am getting used to living on my own, I still think it's not natural. We humans are herd animals. If it were left to me, I'd make us all live in longhouses, like the ones in Nepal, with all the generations packed in together. We've evolved to depend upon each other, we need each other, especially the old. If I were a stone age woman aged 70, I'd never survive on my own. Without the warmth and protection of the tribe around me, the first cold winter would finish me off. But then, if I were a stone age woman, I'd be without the flu jabs and dental bridgework that enable me to boast that 70 is the new 50.

There are mornings when I potter around contentedly at my own pace, watching the sunrise as I sip my orange juice, happy not to have anyone else cluttering up the flat, using up the last tea bag or loo roll without replacing it. Pretty soon there'll be another cataclysm in my life, the arrival of a grandchild. Some claim that then I'll look back on these days alone with nostalgia. Rubbish. I can't wait.

• Esther Rantzen is planning to create a helpline for older people, The Silver Line, to combat the effects of isolation and loneliness.

Sloane Crosley, 33

Sloane Crosley: 'I like being able to come home late and collapse into bed without worrying about waking anyone with my drunken shoe removal.' Photograph: Corbis

Good friends, a couple, are being kicked out of their apartment this month. Decent apartments can be hard to come by in Manhattan, so it's all hands on deck, trying to help with the search.

"I might know of something," I emailed the male contingent of the pair. "What's your budget?"

"We're paying $4,400 now," he shot back.

What a pad one could get for that price!

I sat back from my computer and bristled. Ah, the power of two. There's nothing quite like it. Especially when it comes to paying utility bills, parenting, cooking elaborate meals, purchasing a grown-up bed, jumping rope and lifting heavy machinery. The world favours pairs. Who wants to waste the wood building an ark for singletons? Even the word "singleton", to the American ear at least, reads as particularly insulting. We never use it and thus it sticks out in conversation. Perhaps it's bothersome due to its resemblance to the word "simpleton", which we do use.

I live alone. I have also lived with significant (and sometimes not-so-significant) others for brief periods of time. Truth be told, I was fine either way. There are profound perks and drawbacks to both, too numerous on both sides to list in earnest.

I hope to one day co-sign a lease with another person but, well, it doesn't plague me that I have yet to do so. Put it this way: I've never had to violently tug at my own pillow at 2am to get myself to stop snoring.

In the past, I have not seen the state of my habitation and the state of my love life as connected. This is the nature of being relatively young and living in an urban environment where expensive rental fees can make or break relationships. Cohabitation seems a greater leap in cities because it's all the harder to extract oneself if things turn sour. It's what keeps otherwise functional adults living with their mothers.

The thing is, I am newly single this. For this week (and several more after it, I suspect), living alone feels freshly related to being alone. On top of which, I own a cat. On top of which, I like to eat spoonfuls of almond butter over my sink, put this gross Swedish hair balm in my hair before bed and sleep in old cocktail dresses. None of this was any different when I was romantically teamed with another human, yet suddenly these micro-activities bode poorly as an advertisement for my life.

When I was coupled socially, no one seemed to notice that I was unattached residentially. Two people go out to dinner together, meet each other at shows, take vacations, and suddenly living across town from each other isn't such a big deal. But the building blocks of our daily existence were always separate. He never paid my rent and I never paid his. He was never subject to awkward conversations with my superintendent regarding clogged drains. I was never subject to the etiquette question of tipping his doorman around the holidays. Though most of my friends, attached and not, are in the exact same living situation, society still quietly damns the single-household dweller to one of two diagnoses:

1) Hyper control: I live alone because I am inflexible, intolerant, likely a mysophobic glove-wearer and so stringent about my own schedule that I leave no room for a roommate, lover or a mysterious Italian boarder who happens to moonlight as a DJ.

2) Complete lack of control: with no one to bounce off, my weird behaviours have gone unchecked and my body unshowered. I am socially awkward out in the world while my home is infested with vermin and the crackling sound of broken dreams.

Who among us has not experienced elements of both states? And what does that mean for the future? I wouldn't mind if things were different, but they're not and, truly, I have always enjoyed my space. I love turning the key in the door at the end of the day, being able to decompress, knowing where I left the remote control to the television. I am partial to hot water. I like being able to come home late and collapse into bed without worrying about waking anyone with my drunken shoe removal.

This is not a matter of statistics or trends; it's my life. There is no advertisement for it. Funnily, that's one of the better selling points imaginable: once you realise you're not obligated to persuade others about your existence, it becomes a lot easier to exist.

Sloane Crosley is an author.

Peter Hobbs, 38

Peter Hobbs: 'The mind roams more freely in empty rooms, and the days can spill into evening, and then night, without interruption.' Photograph: David Rose

Even when I've lived with others, I have always been protective of my solitude. I have always needed time to retreat to my own company, and to be alone with my thoughts. It takes me a long while to adjust to sharing living space, to become accustomed to different patterns of noise and movement and sleep.

My first prolonged experience of living alone came in my 20s, when I was suffering from a long illness. As soon as I was able to cope, I moved to live by myself. It was terribly isolating in many ways – I was unable to work or go out – but I wasn't comfortable with company. Illness is a foreign land, and you go always alone. Sometimes I'd go for days or weeks without speaking to anyone, except for brief interactions at supermarket checkouts (in recent years, of course, I would even have been able to find automated checkouts).

It's not an accident that it was during this time I began to write. Gradually, the emptiness of the afternoons began to fill with ideas, and the most pleasurable part of those unhappy days was when I sat down with my thoughts and formed stories, giving myself over to my imagination. Since then, I've always written better when I've lived alone. The mind roams more freely in empty rooms, and the days can spill into evening, and then night, without interruption. Even now I find it hard to write if I know there's someone else in the same building, no matter if they're sitting quietly behind a distant closed door, minding their own business.

Of course the solitude of those years was largely enforced, rather than having been chosen, and though it may have suited my nature, it was a devastatingly lonely time. Something of the pattern of those days has stayed with me, but I try now to monitor my tendencies towards solitude. I'm careful to protect a degree of isolation in my life, but I do not think I will always want to live alone.

I have friends who will live alone for the rest of their lives. They live alone because of choice, or because a partner has died, or because they're so accustomed to solitary living that they're no longer willing to make the compromises necessary for sharing with others. Most of them are content, or at least reconciled to it, but it's clear to me that the happiest of them are those who have arranged their lives so they can spend a great deal of time with as many people as possible.

We're social animals. I think of the way families and friends gather round at times of grief. The way many of us live today can cause the threaded connections of kith and kin to separate and thin, almost to disappear. Yet they reassert themselves in crises. For those who desire it, living alone is a tremendous luxury. But it is a luxury enabled by an existence within technologically advanced, relatively wealthy societies, which insulate us even from the need for others.

Eric Klinenberg is convincing about the hows and whys of the rise in solitary living. The set of circumstances he describes has provided many of us with an extraordinary freedom. I just wonder how fragile they are, and what it might take for us to rediscover how much we need other people.

• This article was amended on 2 April 2012. The original said in the US 27% of households have one person living in them - roughly one in every seven adults. According to research from Euromonitor International 27% of households do have one person living in them, however the one in seven adults relates to 32.7 million people in the US living alone (the 2010 census gives a total US adult population of about 234 million). The original also said the number of people living alone globally increased by 55% in 15 years - from 153 million in 1996 to 277 million in 2011. This has been corrected.