At first sight, Sanya looks much like any other Tokyo suburb: well-appointed homes, supermarkets and fast-food restaurants. In the distance, soaring above the rooftops and mesh of overhead power lines is the unmistakable shape of the Tokyo Skytree.

But its proximity to the ultra-modern landmark is deceptive. Older men in well-worn tracksuits, baseball caps and plastic slippers clutch cans of early-afternoon chu-hi alcopops, and dozens of no-frills hostels advertise rooms with easily the lowest rates in the city – clues to Sanya’s status as a Tokyo neighbourhood like no other, but one that is struggling to adapt to irresistible change.

In Tamahime park “Katchan” – an affectionate abbreviation of his name – is preparing to meet a friend. Nearby, beyond a makeshift tarpaulin home that boasts a flat-screen TV powered by solar panels, men pair up for games of shogi, their chessboard placed on an old filing cabinet. Behind them, a younger man removes all of his clothes and stares into the middle distance.

You won’t find Sanya on any modern maps. In 1966, the government ordered the name Sanya to be expunged from official records

“I’ve come here to drink, chat and enjoy the weather – sunlight is free after all,” says Katchan, a day labourer who moved to Sanya 16 years ago. “I only get very occasional work, but I’ve stayed here because this is where my friends are.”

Katchan is among the last in a line of men who have come to Sanya for hundreds of years in search of jobs, shelter, and a sense of belonging.

During the Edo period (1603-1868), itinerant workers who couldn’t afford regular inns elsewhere in the city took advantage of Sanya’s cheap kichinyado lodgings. After the war, it became home to tent villages for people who had been made homeless by US bombing raids.

The military-issue tents, donated by Occupation forces, gradually made way for wooden hostels. By 1953, around 6,000 people lived in 100 hostels; at its peak a decade later, the day labourer population had reached 15,000 people, scattered among more than 220 accommodations.

But you won’t find Sanya on any modern maps. In 1966, the government ordered the name Sanya to be expunged from official records – it’s now divided into two districts, Kiyokawa and Zutsumi – in an attempt to camouflage its association with poverty, alcoholism, violence and the hiyatoi rodosha – day labourers.

A shopfront and graffiti in Sanya. Photograph: Frédéric Soltan/Corbis via Getty Images

These are the men who built modern Japan in the decades after the war: Tokyo Tower, facilities for the 1964 Summer Olympics, the metropolitan expressway and other infrastructure that lay the foundations for the megalopolis of today.

But the labourers and their home are undergoing changes that promise to transform the area beyond recognition.

Like many other communities in Japan, Sanya is feeling the effects of the country’s skewed demographics. Most of the estimated 1,500 former labourers who live here are in their 60s and 70s, with a small number in their 80s and 90s. As they enter their twilight years, the hostels once occupied by exhausted manual workers have turned into de facto retirement homes for men surviving on meagre pensions and benefits.

While violence is rare these days, Sanya’s increasingly diverse inhabitants are dealing with new sources of tension, says Magokoro Yoshihira, the managing director of YUI Associates, a local community development group.

The dwindling population of ageing labourers has an uneasy co-existence with established families who run factories and other small businesses, together with a rising number of younger people on benefits and foreign backpackers who stay at about 20 of the area’s 140 hostels, attracted by clean and comfortable rooms for a little over ¥2,000 (£15) a night.

Magokoro Yoshihira, managing director of YUI Associates. Photograph: Justin McCurry/The Guardian

While travellers have softened the area’s image, a modest form of gentrification is also underway. Local authorities have long resisted major commercial developments, but they are powerless to prevent private landowners from demolishing their ageing hostels – many of which are considered a fire risk – to make way for more lucrative apartment blocks.

“I can understand the landowners’ thinking,” says Yoshihira, “They believe they can make more money that way. But it’s not just changing the landscape. It means the area is slowly losing its old atmosphere. Sanya isn’t like other parts of Tokyo, so we need to retain some of its original feel.”

There are frequent complaints about noise, drunken behaviour, rough sleeping and, the most common gripe - piles of litter, according to Yoshihira, who says Sanya’s unlikely mix of inhabitants exist in a state of “cold war”.

“It’s turned into an ‘us versus them’ situation,” says Yoshihira, who runs three hotels in Sanya, two for backpackers and one for welfare recipients. Last year, her group opened Sanya Café to encourage residents and visitors to mix over reasonably priced food and drinks. “The cafe’s name is our way of taking back the Sanya name,” she says.

“The former day labourers should be proud of themselves – they’re the men who rebuilt Tokyo, after all. But Japan as a nation has forgotten about them, and people around here are too quick to look down on them. We’re trying to show that diversity is a good thing, and that there’s nothing good about putting other people down to make you feel better about yourself.”

Every Friday, Yoshihira and her colleagues encourage former labourers to join them collecting rubbish in exchange for a meal or drink at the cafe. “We eat together and chat, and it gives everyone a sense of belonging,” she says. “At the same time it’s important for Sanya to prepare for the next stage in its history.”

That future is looking increasingly uncertain for former casual labourers who, forced to leave their condemned hostels, must either find another or be placed in state-subsidised housing, often in distant parts of the city.

Some, like “Aizawa-san” – a Sanya resident, on and off, for three decades – simply end up spending occasional nights sleeping on the street.

The 64-year-old is preparing for an afternoon looking for discarded aluminium cans that he will load into a sack on the back of his bicycle and exchange for cash – ¥100 for every kilogram he collects.

“I don’t have any money left, which is why I have to collect cans,” says Aizawa, who complains that there is little left of his state pension once he has paid for hostel accommodation.

“I’ve seen lots of changes over that time. There used to be riots, but it’s a more peaceful place these days, probably because we’re all too old to fight. Life is hard, but I can be a free spirit here.”

Follow Guardian Cities on Twitter, Facebook and Instagram to join the discussion, catch up on our best stories or sign up for our weekly newsletter