Leah Nelson, 14, says there is no room for argument in her family – literally. "I mean there's no room. I share my bedroom with my three brothers," she says, pointing to a space measuring 3 metres by 2.5 metres. This narrow patch of carpet is where the four Nelson children – Leah, Stephen, 11, Ackeem, nine, and Micah, two – have to dress, study and play. Tumbling out of cupboards are clothes, textbooks and toys.

Leah, who wakes up at 5am to ensure she gets into the bathroom first, is sanguine. She says at least she has a bed of her own – the top bunk. Below her, younger brothers Stephen and Ackeem share the bottom bunk. "I would like to bring my friends round and have sleepovers. But there's no space," she says.

The two-bedroom council flat overlooking a common in south London is home to six people. When the council housing officer visited, he admitted the Nelsons were officially overcrowded. The flat is small: a master bedroom, the children's room, a kitchen and a living room not much larger than the flat-screen television that dominates it.

"It's life's little essentials that we miss out on," says Mona Nelson, Leah's mother. "There's no table to eat on in our flat. The kids have nowhere to do their homework. We cannot buy another bike because we've got two that we keep in the living room." The family's story is becoming a feature of urban life. Shelter says one in four London children live in overcrowded homes – with 391,000 in cramped conditions – an 18% rise since 2008.

The government measures overcrowding using the so-called bedroom standard, which calculates how many bedrooms a family needs according to the number of children and their ages.

The Guardian analysed the most complete public database of the government's housing survey – which sampled 17,000 homes in 2010 – to determine who is most likely to be overcrowded. The lower a family's social status, the more likely it is to be overcrowded. Lone parents have the worst levels of overcrowding and black and Asian households suffer cramped conditions more than white households.

Other research shows that overcrowding can have profound effects for children: underachievement at school caused by lack of space to do homework; illness caused by cramped living conditions; and a lack of privacy leading to depression. In the Nelson household, Leah and Ackeem suffered from chest infections after their bedroom walls became damp and mould began to grow.

Many experts argue that too few homes are being built at a time when the population is expanding. And the homes that are being put up are smaller.

Last year, the Royal Institute of British Architects calculated that the average new three-bedroom home was only 92% of the recommended minimum size – missing the space equivalent to a single bedroom that could comfortably accommodate a single bed, bedside table, wardrobe, desk and chair. The UK, notes Riba, is one of the few western European nations to have no minimum space standards for housing.

With house prices outstripping incomes, many families are cramming into homes that a generation ago would have been thought too tiny to live in. Last year, according to official statistics, 655,000 households were living in overcrowded conditions – 3% of the population – up from 530,000 five years ago.

It is homes in the social sector – owned by councils or housing associations – that are the most packed. Families living here, inevitably poorer, are seven times more likely to be cramped than those who own their own homes.

The government says it will ease overcrowding for those on the lower rungs of society with market mechanism – a mix of coercive policies that cut benefits if people do not respond to them. The first idea is to cut the benefits of those "under-occupying" social housing if they do not move to smaller properties.

The second will cut housing benefit to ensure only those who work can afford to stay in the private rented sector.

The government's impact assessment admits the under-occupation policy will have the smallest impact in London, where the problem of overcrowding is most intense. And many experts say that a downturn coupled with welfare cuts will only exacerbate the problems faced by the poorest in society.

Last month University College London's Institute of Health Equity warned that government welfare reforms "will make it harder for households ... to cover housing costs. Adequate housing may be more difficult to afford during an economic crisis and households may be forced to live in environments that may constitute a risk to health, such as homeless situations, overcrowded housing, and housing in a poor physical condition".

In Southwark, south London, Mona Nelson, 35, would like to move out of her small flat – preferably to a four-bedroom home nearby as Leah needs her "own space". She says with the children settled at school and family on hand to help, they do not want to leave the community. Buying in London is out of the question.

In reality she has two options: to wait for a council house to turn up or to try the private rented sector. However, a shortage of council properties means that it will be years before she can move. Private renting, she says, is impossible because the gap between "rents and housing benefit cannot be made up by wages". She says she should have moved before the government caps were announced.

The problem for Nelson, who juggles being a business student with childcare, is cash. Her husband works at a garage full-time. Including tax credits, earnings and benefits, the family have about £25,000 a year to live on. The council rent on the two-bedroom flat is affordable at £250 a month and the Nelsons are entitled to about £20 a month in housing benefit.

Renting in the private sector would, says Nelson, be impossible. Rent for four-bedroom properties in Southwark starts at around £1,700 a month, but the government has capped housing benefit at about £1,600 a month.

"That's if you can even get a cheap property and if someone rents it to you on benefits. More likely you get an expensive one and the real problem is that rents are going up while benefits are not. So you will find yourself being priced out of the housing market pretty soon."

When they first moved in a decade ago, the flat was perfect for a couple. Even with two children it was fine, but now she says it is "just too small".

"I have heard government ministers saying children are your responsibility but my last two were unplanned. What are they suggesting? I think the government needs to think about capping private rents, not capping benefits."

Peter Ambrose, visiting professor in housing studies at Brighton University, says one big issue is how to quantify overcrowding.

"The legal definition goes back to 1935 when it was acceptable to sleep in kitchens," he says. "Without an updated measure there's no impetus to measure the effects of overcrowding or who is overcrowded. It's maddening."

Ambrose says his own research in Wandsworth, south London, for London Citizens in 2010 found that more than half of the families of schoolchildren interviewed in the borough raised concerns over lack of sleep and tension because of overcrowding.

He found that despite England having smaller useable floor space in its homes than any other western European nation apart from Italy, there appeared a "lack of interest" in Britain to assess the design and size of real houses and the importance of community ties.

However there is increasing evidence that overcrowding is spreading into previously untouched parts of Britain. Becky Turner, a teaching assistant, 26, lives with her partner, a civil servant, and three children in a two-bedroom, privately rented home in Christchurch, Dorset. Her two boys, aged four and seven, share and the two-year-old sleeps in her parents' bedroom. However within a couple of months the parents will be sleeping in the living room – as the toddler will need her own bed.

"We can't afford to move. At the moment we get £240 a month in housing benefit and the rent's about £600. But you cannot get a three bedroom place for anything less than £900 and the housing benefit won't cover that. So we are on the waiting list for social housing … along with 2,500 other families in Christchurch.

"My boys share a small bedroom. There's no room for them to play. Our youngest is just a baby and she shares a bedroom with my partner and me. A full night's sleep is impossible, if one's awake we're all awake. This has a knock-on effect with the children's education and our abilities to do our jobs."