The first time I met Ian Dick, the head of private housing at Newham council in east London, he took me on a walk to look for “beds in sheds”. It was 2011, and alongside criminal levels of overcrowding in private rental properties, there was a growing problem of people living in illegal structures in back gardens. It was not uncommon to find 10 or 20 people living in a room above a fried chicken shop, in a basement, or in ramshackle outbuildings. When we met again, five years later, he was happy to talk to me, not because these problems had disappeared, but because he was proud of the council’s private rented sector licensing regime. Introduced in 2013, it was the first such scheme in the country and had led to 800 prosecutions and 28 landlords being banned from renting property to tenants.

This time we met in Forest Gate, traditionally one of the most deprived parts of Newham. “This is an area undergoing the most dramatic change – the council doesn’t use the term ‘gentrification’, they use the term ‘regeneration’,” he said as we strolled down a pleasant high street in the sunshine, looking up at Victorian facades renovated by the council. Along the road, hipster cafes and pubs were interspersed with clothing retailers, halal butchers and phone shops. To show me the reality in some of the flats above, he took me around the back, where an entire street was accessed by a badly maintained private alleyway, with a huge pile of mattresses dumped at one end.

The mattresses were outside a property where, until the previous week, 20 people had been living in a makeshift structure in the yard outside. “The landlord is under caution, he’s not arrested, he’s been housing people desperate for accommodation – people were paying £400 a month to stay in this,” Dick said. Further down the back alley, the entrances to a large number of flats were up dangerous-looking fire escape stairs, particularly hazardous for small children. In one of the properties a scooter stood at the foot of a rickety open stairway.

At the back of a large Victorian house, we were met with a smell of leaking sewage, and the once-white walls were now filthy and soot-stained. “This must be illegal. That’s the thing about English housing – there’s little that’s illegal but there’ll be breaches here,” Dick explained, telling me that English property law does not target property standards, but that interventions can be made around public health. In this case, the breaches would be drainage and conditions detrimental to health.

We paused to look at the 'to let' signs in the newsagents. One offered a room share for four people for £160 a month

“This is the sort of private rented sector that still exists in 2016, even after all we’ve done,” he said, noting details down in his notebook so that his officers could visit. The interior of one of the flats was visible through a grim-looking security gate at the back door. Through the bars I could see a toddler and his mother. This was no place for a child – or anyone – to live, but it was also obvious that once they were evicted, their fortunes would not necessarily improve. There would be nowhere to go, and even if they qualified for social housing, they would most likely enter the world of substandard temporary accommodation.

Walking back down the high street we paused to look at some of the “to let” signs in the newsagents’ windows. One in particular stood out, offering a room share for four people for £160 a month per person: effectively a bed space in what may or may not be legal accommodation. Renting bed spaces, I had already heard from a number of sources, is becoming more common, with bunk beds visible in the front rooms of nearby terraces. While we sat having a coffee, Dick explained this was partly down to the increasingly common – and highly profitable – practice of what is known as “rent to rent”.

“An estate agent will rent a house to an individual who will then let it out to others, who might also sublet. There’s a whole submarket in rent to rent. It’s all done with no documentation, they don’t ask for references, and when we go round we might find 15 people there. “There’s different levels of this going on – everybody has to share because nobody can afford to rent,” he said, adding that sometimes criminal gangs might rent 20 properties, then re-rent them at a 20% higher price. The council had prosecuted 25 agents, he said, and pointed to how to find them through ads in the local papers: “Where you see agents offer ‘guaranteed rent’, there are no controls on lettings agents.”

Adverts on hoardings all over London show a city populated by smart-looking people in luxury balcony apartments. The city is the destination of choice for foreign investors and the oligarchs, billionaires and super-rich who make up the so-called “alpha elites”, all of whom are attracted by the UK’s favourable tax environment. Entire neighbourhoods – St John’s Wood, Highgate, Hampstead, Notting Hill Gate, Kensington – have changed out of all recognition in the past decade. Estate agents refer to these centrally located, “super prime” areas as “golden postcodes”. They have long been wealthy places, but in the past, like most of London, they were also mixed. Now ultra-high-net-worth individuals have displaced even the wealthy from Kensington; they in turn displace others to suburban areas, creating a domino effect that ripples out through the city, with the consequence that average-income earners and the poor move to the periphery or out of the capital altogether, which places pressure on housing and pushes up prices around the country.

London’s skyline is being transformed by one of the greatest waves of new construction seen in the city, with plans for no fewer than 300 luxury residential towers. From Nine Elms up to Vauxhall and along to Southwark and Blackfriars bridges, mile upon mile of apartments in gated complexes have already been built. At Elephant and Castle the Australian property developer Lendlease is working with Southwark council to render the area unrecognisable, replacing the affordable housing that once characterised it with a forest of gleaming towers.

There is a direct link between the wealth of those at the top and the capital’s housing crisis – which affects not just those at the bottom but the majority of Londoners who struggle to buy properties, or pay extortionate rents. The 2008 financial crash created a new politics of space, in which people on low incomes are forced out of their homes by rising rent and the wealthy are encouraged to use property for profit. These trends are not limited to London. The same currents of global capital are also transforming San Francisco, New York and Vancouver, European cities from Berlin to Barcelona and towns and cities in the UK from Bristol and Manchester to Margate and Hastings. This isn’t gentrification, it’s another phenomenon entirely. Global capital is being allowed to reconfigure the country.

Stop spending money on avocados? Good idea, I’ll have a house deposit by 2117 | Rhiannon Lucy Cosslett Read more

Public housing accounted for a huge proportion of British accommodation throughout the 20th century. Since the 1980s, public housing stocks have been steadily eroded, through the combination of right to buy, which saw the sell-off of 2m council homes, and buy to let, which has resulted in 40% of those former council homes now being owned by private landlords, who rent them out for three and four times the amount that council tenants pay for the remaining local-authority-owned properties.

During the 1980s a decision was taken to cease building housing for those on lower incomes and to create instead a system whereby housing benefit would “take the strain”, rising to cover increases in rents. At the time it was predicted that the change in policy might lead to higher rents but nobody believed they would rise to the level they are today. The combination of right to buy and the decision to stop building council housing shifted social tenants into private rented housing. Research from the National Housing Federation shows that the amount of housing benefit paid to private landlords doubled from £4.6bn in 2006 to £9.3bn in 2016.

The shortage of affordable housing has given rise to a range of problems in private rented accommodation, from slum landlords and “beds in sheds” to middle-class Londoners under the age of 45 who can no longer afford to live in the city. A generation is being affected and our essential services, such as hospitals and schools, and the majority of our small and medium-sized businesses, are being undermined.

“We think broadly a third of landlords are well-meaning and do a good job,” said Southwark councillor Mark Williams, “a third are well-meaning and do a bad job, and a third are rogue landlords. At the fringes of that, you have slum landlords and criminal human trafficking.” Southwark, in south-east London, despite increasing areas of high-cost development, still includes large pockets of poverty. Williams told me that the council found a two-bedroom flat on the Aylesbury estate with 20 occupants, who were being bussed down to Bromley, 40 minutes away, every day to work in a sweatshop.

Facebook Twitter Pinterest Overcrowded rented housing in Newham, east London. Photograph: Newham council

Since Newham introduced its licensing scheme, the council has put a maximum limit on the number of people allowed to share one room and has stipulated that agents must display what that limit is, although some don’t. “Lots of them were non-compliant at the beginning but when we looked at them they started to comply, and then we relaxed and they went back to what they were doing before,” Dick said. It doesn’t surprise him. “Why wouldn’t they? The housing market is broken and the conditions are ripe for exploitation. We’ve got the worst of both worlds – a market which is controlled but not regulated. Most people we’ve prosecuted through the licensing scheme are criminal landlords, although the government prefer the term ‘rogue’.”

Although it is notoriously difficult to get accurate figures, a 2013 report by the Migrants’ Rights Network concluded that Ealing may have as many as 60,000 occupants in illegal structures, and Slough borough council, which deployed planes equipped with thermal imaging equipment in an effort to spot them, may have as many as 6,000 beds in sheds. In 2013 a BBC investigation found estate agents renting out beds in sheds in Willesden Green and Harrow.

Newham’s licensing scheme has been widely praised, and many councils have expressed wishes to emulate it. But in 2015, to the delight of landlords, the government made it clear it did not want to extend the scheme’s use, with then housing minister Brandon Lewis describing licensing as a “tenants’ tax”.

Even though the Department for Communities and Local Government had given Newham £1m to support the licensing scheme, little-known changes to the law have seen some forms of licensing extended, but also made it much harder for councils to introduce licensing in the first place. The new law ensures that councils must now seek permission from the secretary of state for any licensing scheme that would cover more than 20% of their geographical area, or affect more than 20% of privately rented homes.

In Redbridge, north-east London, the council’s application for borough-wide licensing was refused by the secretary of state. As a result, council leader Jas Athwal said it would be “impossible for us to … deal proactively with poor standards of rented accommodation”. It is clear, then, that an effective means of clamping down on rogue landlords has been found – the government just refuses to support it.

Extreme overcrowding is illegal, but poor conditions are also common in the “mainstream” private rented sector. Jan is a university graduate with a good job, who lives in what she describes as “middle-class poverty” despite earning close to £40,000 a year. She and her husband don’t have enough money to make ends meet, even though they both work full time.

The family last moved three years ago after their landlord put their rent up by £450 a month. He wanted to sell to a developer who planned to turn the flats into one luxury home, and needed the tenants out. Conditions where they were living were far from ideal: the heating didn’t work properly and Jan and her partner slept in the front room so that their two children, a 10-year-old boy and seven-year-old girl, could have their own rooms. That isn’t possible in the tiny two-bedroom flat they are in now, so she is thinking of putting a mattress in the alcove in the hallway for her daughter. “I’m going to put a curtain up and have her in that, but when I think about that, it’s Dickensian,” she told me. Revealing how low her expectations have sunk, she added: “We’re in a flat which is substandard accommodation, but what we love about it is that the heating works.”

The other advantage is that the overcrowded, moth-infested flat is cheaper, but it is in the wrong catchment area for the secondary school her son has applied to, which is causing him great anxiety. An estate agent has found Jan’s family a larger flat in West Norwood, where they used to live, but that would, as before, eat up two-thirds of her salary. “We can stay here and clear our overdraft or go back to West Norwood, have a kitchen diner and live totally beyond our means. It’s these ridiculous choices,” she explained. The other option would be moving to Croydon or Mitcham, but prices aren’t much lower there. “We might be able to get a semi with a garden but the kids would not be in the catchment area for their schools, there’s a massive commute and you lose all your networks.”

Jan is paid far above the average salary and her partner works full time as a teaching assistant. But after deducting the cost of childcare, they have nothing left to spare – “it’s second-hand clothes and no holidays. They call it ‘the squeezed middle’ but it’s poverty. It impacts on a whole range of things you can’t begin to anticipate,” she said.

To cap it all, on the day we met she’d had a terrible journey into work with cancelled trains and her Oyster card had charged her an extra £20 by mistake. “I just stood on the platform crying. It’s beyond my capacity to cope with any of it,” she told me as we sat in her office.

For the past 30 years, Britain’s economy and culture have been built on the Conservative ideal of home ownership. But, despite all the rhetoric surrounding this issue, home ownership in Britain has not exceeded the European average of 70% since the early 2000s. In fact, it has fallen to 64% – the lowest level in 30 years. And in 2017 private renting overtook mortgaged home ownership in London.

There are more requirements to run a cattery than to rent out a home Betsy Dillner, Generation Rent

“This is a middle-class issue now,” Betsy Dillner, director of Generation Rent, the campaign group for better private renting, told me. “People think we represent this middle-class professional group, but if you can find a way of making the private rented sector work for the most vulnerable people in society, then it will work for everyone.” Today, 11 million people in Britain rent privately in an overlapping series of sub-markets, ranging from slums to luxury housing.

As an organisation, Generation Rent would like to see rent controls, a national register of landlords and fully licensed lettings agents – all of which Dillner admits are highly unlikely in the current ideological climate. “There are more requirements to run a cattery than to rent out a home. There should, of course, be regulations to look after pets, but I’d like the same rights to be afforded to the place we call home,” she said.

Like 2.3 million other Londoners, the Green party London Assembly member Sian Berry is a private renter, and has shelled out more than half her pay in rent and lived in six different houses since she moved to London 20 years ago. In 2016, she published the Big Renters Survey of more than 1,000 renters. According to those surveyed, rising rents were the most common problem experienced, and seven in 10 renters suffered from repairs and maintenance not being done. Damp, mould, broken boilers and dangerous electrics feature prominently in the responses, with many reports of landlords who evict tenants rather than pay for repairs.

Facebook Twitter Pinterest The type of accommodation known as ‘beds in sheds’, in Brent, north-west London. Photograph: Brent council

One said: “After thousands of pounds’ worth of electrical items all broke down at once, an electrician assessed our flat. The wiring was the worst he’d ever seen and was, in his words a ‘death trap’. We were living there with a newborn baby. Our landlord chose to evict us, making us homeless, rather than carry out repairs.”

Another responder reported that they had to wear rubber gloves in the shower to avoid electric shocks and a third wrote: “Once a ceiling of an old house fell on me and all my stuff. I let my landlady send workers into the house to fix it for two months, during which time we still had to pay some rent. Immediately afterwards she put the rent up to a level we could not afford and we had to move out.”

Yet another factor placing great pressure on the private rental market is the growth of companies such as Airbnb. By allowing owners to rent properties on a night-by-night basis to affluent visitors, their business model removes accommodation from the longer-term rental market while keeping purchase prices high. As a result, Airbnb has become a focus for housing activists across Europe and in the US, particularly in cities that attract large numbers of tourists such as New York, Barcelona, Berlin – and London. In New York, Barcelona and Berlin, city governments have restricted owners’ ability to rent out whole properties through Airbnb. But while oversight is being introduced in some places, it is a notoriously hard business to regulate, particularly when there are significant profits to be made.

The bedroom tax, introduced in 2013 for occupants of housing association or council property, presents a mirror image of Airbnb. In a society where the idea of public housing has collapsed, a financial penalty is imposed on people in social housing with a spare room, while those who are lucky enough to own a house with extra space find themselves with an additional source of revenue.

Another feature of the new economy in private renting is property guardianship, where developers offer lower rents to people prepared to live in properties due to be redeveloped. Often these are in edgy locations, such as the Ernő Goldfinger-designed Balfron Tower in east London, where groups of artists move in on a temporary basis, taking advantage of the subsidised accommodation costs and bringing cachet to the area. In the past, artists squatted abandoned buildings in places such as Shoreditch in east London and Brixton in the south, creating communities that have long been credited with maintaining the city’s diversity while also setting in motion a more manageable process of gentrification. In 2013, however, legislation was passed that made squatting residential buildings illegal, with a maximum penalty of six months in jail and a fine of £5,000. Property guardianship can be seen as an attempt to co-opt the “desirable” elements of squatting by young, creative people in a blatant bid to accelerate gentrification. At the same time, it is a highly lucrative, unregulated business, and “guardians”, who have no property rights, often have to put up with poor conditions and can be evicted at short notice.

The major concern for the government and employers in London is that people who do not earn enough to meet extortionate rents will leave, hollowing out the city and threatening its labour market and culture. “We see this with employers saying they’re having a really hard time retaining professional level jobs, let alone cleaners. London is losing teachers – they’re commuting from Luton and they’re giving up – it’s having a massive knock-on effect,” Dilner said.

Facebook Twitter Pinterest An outbuilding in east London. Photograph: Barking and Dagenham council

The vacancy rate for nurses at London’s hospitals is 14-18%, according to a report from the King’s Fund thinktank, and the number of entrants to teacher training has fallen 16% since 2010, according to Ofsted. But it’s not just carers, nurses, teachers, artists and university lecturers who can’t afford to live in London. Fifty Thousand Homes is a business-led campaign group – including the Royal Bank of Scotland, the CBI and scores of London businesses – formed to push the housing crisis up the political agenda. Its research shows that on current trends, customer services and sales staff at almost every level are being pushed out of the capital. Three-quarters of business owners believe that housing costs are a significant risk to London’s economic growth and 70% of Londoners aged 25 to 39 report that the cost of their rent or mortgage makes it difficult to work in the city.

Vicky Spratt is a 28-year-old journalist who worked as a producer of political programmes at the BBC but left because she felt the issues affecting her generation, such as the housing crisis, were not being covered properly. “A lot of issues were dismissed by the older generation – it didn’t affect them. They all owned their own homes,” she told me. Spratt joined the digital lifestyle magazine The Debrief, aimed at twentysomething women, and began an online petition against lettings agents’ fees that gathered more than 250,000 signatures. As a result, Philip Hammond, as chancellor of the exchequer, banned lettings agents’ fees in his 2016 autumn statement – the biggest victory for private renters to date.

Spratt describes herself as a reluctant campaigner, but her circumstances pushed her into it. She currently pays £1,430 per month, not including bills, for a one-bedroom flat which she can afford because she shares with her boyfriend, but she used to live in a room “which was literally the size of a bed”. “The walls were very thin because it had originally been part of one room, which the landlord split into two. I noticed after about six weeks my mental health deteriorated. If I wasn’t in a relationship I would be looking at going back to that,” she said. Spratt earns enough to get a mortgage but, because rents are so high, not enough to save for the 20–30% deposit required.

“The common thread for people my age is that we don’t own our own homes and potentially we never will. The housing crisis is older than me and it shocks me that nobody did anything about this, and I want it on the news agenda,” she said. “This is structural neglect. The buy-to-let boom and the unregulated market have a lot to answer for.”

Spratt believes that a key reason for this political failure is that “people who don’t experience this issue are in charge”, pointing out that almost a third of MPs are buy-to-let landlords. An Oxford graduate who has worked in Westminster, she also suspects that a factor behind the success of her campaign is because “I look and sound like them”.

Betsy Dillner, director of campaign group Generation Rent, said: “This is the neoliberal free market ideology which has been alive and well in the US for a long time and it is certainly taking over here.”

Dillner fears that the outcome will be that the city will become soulless, “a place where you can’t raise families, be young, start up new businesses – all those things that make the city we love – housing costs will suck the soul out of that.”

While population shifts to other towns may seem like a good way of rebalancing the wider British economy, the problem is that with nearly every FTSE 100 company headquartered in the capital, the majority of jobs are there too.

It is not possible to contemplate real solutions to the housing crisis without profound structural economic change. A weak private sector in the regions and public-sector jobs shrinking rapidly as a result of cuts mean that many people still look for work in the capital. And while some will leave the city, others will put up with otherwise unacceptable conditions in order to keep their jobs and family ties. This means that generation rent must pay through the nose to live in an increasingly sterile city – a playground for the rich in the centre, surrounded by gentrifying hipster hinterlands, and substandard housing for cheap labour in places such as Barking, Dagenham and Edmonton.

Facebook Twitter Pinterest The exterior of a rental property in Brent. Photograph: Brent council

Expecting people to move each time they are priced out of an area brings its own problems. Moving house is rarely an easy experience, even when the move is voluntary. But it’s incomparably worse when it involves being wrenched away from the support networks, daily routines and the sense of identity that comes with being able to call a place home. Even the threat of being uprooted, known as housing insecurity, is a significant contributor to mental health problems.

A secure and stable place to live is the most important factor in personal wellbeing, according to a study by psychiatrists Ciaran Abbey and TBS Balamurali. The study, titled “Housing the Mind”, cites evidence that prolonged periods in temporary accommodation can adversely affect a person’s mental health – and so can spending more than 30% of income on housing. “When a disproportionate amount of income is spent on housing, this leaves people less able to purchase other necessities such as adequate food, increasing the family’s vulnerability to disease but also the anxiety and sense of helplessness that results when unable to make ends meet,” says Abbey. When individuals lose their homes, they can lose not just physical shelter but their entire world.

This is not just about building more houses, it’s about the kinds of houses that we build and who we build them for. It is also about our priorities as a society. Property is being built in London – for wealthy foreign investors instead of the people who need homes most. As the stock of social and genuinely affordable housing dries up, rents are soaring and the taxpayer is being forced to contribute billions of pounds in housing benefits for poor-quality private lets. So far, the proposed solutions from both main political parties involve only tinkering with the existing situation. Instead, we need a new social contract that ensures housing is once again viewed as a right for all, not just an asset for the few.

This is an edited extract from Big Capital by Anna Minton, which will be published 1 June by Penguin

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.

• This article was amended on 26 and 31 May 2017 to remove a reference to metal spikes on the pavement in Southwark. It was subsequently confirmed that the spikes have been removed.