Britain is in the midst of a housing crisis, but has more than 260,000 empty homes. How on earth do houses such as 20 Hayles Street – a council-owned property in a city ravaged by homelessness – stay unoccupied?

Twenty Hayles Street looks unloved rather than abandoned. Its windows are dirty but not yet boarded up. Net curtains partially obscure the front room, where John Bailey remembers his Uncle Ernie playing Frank Sinatra on a gramophone.

“Aunt Bette would be out here sweeping the front step at least once a day,” Bailey, who is 66, says on the pavement near Elephant and Castle in south-east London, where he was born and still lives. Today, the step is covered in dust. A landslide of junk mail threatens to creep out from under the door. “She would clean this place top to bottom every week. It was like a palace in there.”

In 1916, Bailey’s grandfather, an upholsterer, moved into No 20, which was built in 1853. For almost a century, the four-bedroom home was the noisy heart of the Bailey family, which would grow to occupy a dozen addresses on the road. But, after Bette got cancer in 2015 and moved to a nursing home, John is the last Bailey on the street.

The retired community nurse strolls from his house towards the Prince of Wales pub, where his aunt would gossip over her daily port and brandy. “That one’s empty,” Bailey says, pointing at another house. “That’s empty,” he adds, two doors along. “And that one … and that one.”

On a short street in London’s travel zone one, where residential skyscrapers rise as symbols of the city’s affordable housing crisis, four family homes stand empty. Three are owned by Southwark council, which has 11,000 families on its waiting list for social housing. They have been “voids”, as the council describes them, for more than six months, the government’s definition of long-term empty. No 20 has been empty for more than two and a half years. The fourth house, which can be traced to an Italian commodities trader, has been empty for almost a decade.

“I think it’s criminal,” says Bailey, who rents from a housing association (privately owned houses on the street frequently sell for more than £1m). He catches up regularly with his family, most of whom have left London for Essex, Bedfordshire and beyond. Bette died soon after leaving the street. “They always say, ‘How’s 20?’ and I tell them it’s still empty. They can’t believe it – after all this time. What on earth is the reason for it?”

Facebook Twitter Pinterest John Bailey outside 20 Hayles Street. Photograph: Graeme Robertson/The Guardian

From the “buy-to-leave”, foreign-owned mansions of Kensington and Chelsea to the crumbling inner-city terraces of former industrial towns and cities, empty homes inspire a sense of bafflement and injustice. Vince Cable, the Liberal Democrat leader, called it a “national scandal” that more than 11,000 homes in the UK have been empty for more than 10 years, according to figures his party published last month. The number of homes empty for more than six months has gone up in Scotland and Wales to about 37,000 and 23,000 respectively, according to separate government figures. They still stand at just over 200,000 in England, despite an overall decline.

Often the reasons for empty homes – privately owned or social – are simple. Residents die and probate drags on. Landlords can be slow to carry out repairs. Optimistic buyers fail to secure funding or planning permission for work, or find that properties they had planned to sell quickly gain value more slowly than anticipated. Greed, bureaucracy, bereavement, neglect and naivety can all play a part.

But what Bailey finds hard to fathom is why council homes can stand empty for so long during a housing and homelessness crisis. There were more than 24,000 council homes empty for more than six months in England in 2016, including more than 6,500 in London, and almost 600 in Southwark, the borough that includes Elephant and Castle.

“Every week I receive a call from someone in desperate need of a home,” says Maria Linforth-Hall, a Lib Dem councillor in the Labour-run borough. Last year, she helped organise a petition to put pressure on Southwark to do more to fill homes on Hayles Street and the surrounding roads. Residents worry about burglars, as well as unsightliness and the knock-on effects of dereliction. “I’m still waiting for a proper reply,” she adds.

“We appreciate that some of our voids take longer than we would like to fill,” says Stephanie Cryan, the deputy leader of Southwark council and the cabinet member responsible for housing. She says the council has now formed an empty homes team. “Contractors and their capacity” are part of the problem, she adds. But the fate of the Hayles Street “voids” reflects a national trend. The council is preparing only one of its three empty houses for a new social tenant, after work to tackle damp. The other two will be sold at auction, with the proceeds going back into the council’s housing fund.

Cryan says the council meets to decide whether to sell empty houses worth more than £750,000 if refurbishment is too costly. “We’ve had houses where it would have cost over £150,000,” she says. The council says it has sold 60 empty homes this way in the past five years. In the case of No 20, Cryan says that, despite work two years ago to install a new kitchen and bathroom, fixing the damp and the roof would cost more than £50,000. She expects the house to sell for more than £850,000.

Across the country, councils are selling homes almost three times faster than they can build them. Figures provided to the Lib Dems by 72 councils last year also showed that the gap is widening. In the worst cases, local authorities sold 20 times the houses they built. Southwark says it is “roughly on target” to build the 1,500 homes it promised in 2014 to complete by the end of this year. But last year, it admitted it was on course to complete just 400 of these. In the five years from 2012 to 2017, meanwhile, it sold 1,312 homes, almost all under the right-to-buy scheme.

Right to buy is the biggest threat to housing stock, local authorities say. In 2012, the government increased the discounts offered to tenants, compelling councils to sell homes at almost half price. This triggered a “fire sale” of council houses, the Local Government Association says, of more than 13,000 in 2016/17, up from 2,638 five years earlier. The scheme permits councils to keep just a third of the discounted sale price, the rest going to the Treasury. A cap on the amount councils are allowed to borrow to build new homes further dents their ability to meet demand. And they fear this will get worse under Conservative proposals to extend right to buy to housing association tenants, which local authorities would be compelled to finance by selling high-value homes. Auctioning houses when they become empty, and reinvesting all the proceeds, would appear to be a better bet, but Southwark insists selling is always a last resort. “There is a housing crisis and we want to keep our stock,” Cryan says.

Facebook Twitter Pinterest Photograph: Graeme Robertson/The Guardian

Hayles Street was always intended for working-class Londoners. The area was developed in the 18th century, during the urbanisation of south London. Later, the church-run Hayles Charity built on fields acquired from Robert Hayles, a haberdasher. The charity leased the land to developers to support poor children, before taking back the new houses to use as affordable homes.

No 20 is part of Hayles Terrace, a row of seven houses built in the mid-1850s. In 1857, a notice in the Morning Advertiser, posted by Mr W Arundel, a coal merchant living at no 14, offered “three newly built houses – let to respectable tenants”, at £22 a year, or to buy for £160 each with 58-year leases. Censuses show who lived at No 20 before the Baileys arrived in 1916, by which time the charitable Hayles Estate was letting the house: a bricklayer in 1861, then “crack painters”, a stonemason and a glass finisher. In 1911, four families shared the house.

But in 1978, the charity sold the entire estate, which also included the next street over, to the council for just £750,000 (about £4m today). After right to buy, and several transfers to housing associations, the council today has 24 tenancies on Hayles Street, about a third of the road’s homes.

Southwark sold the fourth empty house in 2009. M&A Real Estate bought it for £355,000. It is now worth more than double that, yet lies empty, a CCTV camera pointing at its reinforced door. “An empty house is not good for anyone,” Alessandro Pacciana, the company’s director, tells me on the phone. The Italian trader has lived in central London for 20 years. “If you have a Ferrari that you never use, sooner or later the battery will get ruined, the tyres will go flat.”

Pacciana says he planned to build an extension before letting the house, but then fell out with his builder. “I was going to have the work restarted but with Brexit and all the other issues in the past few years, things have slowed down,” he says. “It’s an ongoing process.” He says he is setting aside funds to finish the job, and won’t sell beforehand. He insists he is not sitting on the property purely to double his investment.

Councils must contact the owners of empty private homes. They can award refurbishment grants or refer owners to social enterprises or housing associations, which can do the work and let the homes. Beyond that, authorities can use empty dwelling management orders to force an owner to let a “nuisance” property. But councils have faced criticism for rarely using the power. Southwark says it prefers to work with landlords.

In the north of England, where the empty homes problem is acute, solutions can be extreme. The Pathfinder scheme, introduced by the Labour government in 2002, was intended to revive former industrial towns and cities, but its failure – and cancellation by the coalition government in 2010 – left many councils with decrepit homes they could neither restore nor sell. In a desperate attempt to fill them, several councils are selling houses for a pound each, perhaps most famously in Stoke-on-Trent.

Play Video 8:24 £1 for a house: made in Stoke-on-Trent

In Toxteth, Liverpool, a housing enterprise is working with the council on the latest phase of its redevelopment of Welsh Streets, where empty terraces served as a location for gangland period drama Peaky Blinders. When the scheme is finished, Placefirst will have redeveloped and let out 294 houses there, many originally acquired for next to nothing. It already has 300 tenancies in homes it has restored or built across the north of England. It rents at fair market rates to families who do not qualify for social housing and would typically end up in overpriced, neglected rental homes. “They’re junior doctors, barbers, dentists – people without inherited wealth living month to month,” says David Smith-Milne, the company’s founder and managing director. The former management consultant, who grew up in a council house in Runcorn, Cheshire, says empty homes “are symbols of successive governments totally cocking up the housing market. They illustrate the way housing is now a financial commodity rather than a fundamental need.”

A week after Bailey looked despairingly at No 20, an auction sign appears above the front door. The house will be sold this week at a central London hotel. Not long before the sale, Bailey jostles for space during a viewing. The rooms are bare and scrawled with a surveyor’s marker pen where damp climbs the walls.

It is the first time he has been inside the house since Bette left it. He plans to go to the auction with his cousin Janet, and fears the house his family called home for a century will now fall into the hands of developers. “I’ve been buying lottery tickets for the first time,” he says. On his way out, he picks Bette’s post out of the pile of estate agent flyers, and takes it home.