On the night of the EU referendum, photographer Andy Martin sat at home as the results flooded in. He didn’t usually watch television, but last June he kept the news on with the sound down. His home city of Sunderland prides itself on being the first to declare in a general election – but that night was different. Gibraltar, Newcastle upon Tyne and Orkney had all announced their results before, at 12.15am, Sunderland finally declared it had overwhelmingly voted to leave the EU.

The following day, Martin again walked around the city in which he was born in 1984, and where he has spent almost all his life. Sunderland’s urban landscape has been his principal photographic subject for more than a decade, and the general mood that morning was unusually buoyant.

“I felt like a stranger in my home town,” Martin says.

Soon afterwards, he decided to leave Sunderland. First he took a long holiday in Europe, driving his van 4,000 miles around France, Switzerland, Germany, Belgium and Holland. By the time we meet in his photographic studio, he is about to move to London to start a new job.

The walls of Martin’s studio are lined with ghostly monochrome portraits of Sunderland’s present-day inhabitants – members of bands, artists, his friends: a collection of photos he calls his “census” of the city. These portraits contrast with his urban photos which are usually unpeopled, but share a similarly haunting atmosphere, as if he knows he is capturing something that will soon be gone.

CCTV screens in Gillbridge police station before it closed in 2015. Photograph: Andy Martin

Sunderland was shaped by the rapid expansion of heavy industry in the 19th century. When a seam of high quality coal was struck in 1834 at Wearmouth Colliery, it became a boom town for mining. Shipbuilding thrived, and by 1901, one in every five employed men in Sunderland worked in the industry; they were the best paid workers in the city.

Workers – economic migrants, if you will – travelled to take up jobs here, and its population grew from 26,000 in 1801 to 182,000 a century later. The city expanded to accommodate these new inhabitants, with terraces of workers’ cottages constructed within walking distance of the docks and colliery.

After the depression of 1907-8, however, shipbuilding went into a decline that ultimately ended in 1988, when the last ship on the river Wear was built. Coalmining at Wearmouth Colliery continued until the end of 1993; now it’s the site of Sunderland AFC’s Stadium of Light.

Martin’s studio is in an old warehouse near the river, reached by walking along dark streets past vacant lots, industrial units and empty pubs, then taking a sharp left at the Liebherr crane factory. The nostalgia for lost industry is palpable in Sunderland, and physical traces of that era can still be found, especially along the river, with old buildings repurposed or empty, earmarked for demolition or already gone.

The Turbine Steam Ship Manxman in Pallion Engineering’s dry dock in Sunderland, 2007.

As a child, Martin would hear stories of the city’s industrial past – of the shipyards and coal mines. As an adult, he felt compelled to photograph these industrial spaces, and to picture “what they were like when they were inhabited … just imagining the shipyard buzzer going off and everyone dashing out at five. Almost to breathe a strange bit of life back into them.”

Photographing Sunderland at night became an obsession: “I just wanted to portray it a little bit differently,” he explains. “I think I’ve always been a night-time person.”

In the dark, Martin says, the city felt like it was changed somehow. At night, “you become very aware that there’s nobody there. It’s a very lonely place. I think a lot of the time the photos are just trying to reflect that.”

He also liked the way film would capture night scenes when it was exposed for long enough to absorb the glow of available light. He would normally go out at 10 or 11 at night, returning home three or four hours later.

In a report published earlier this year, the Joseph Rowntree Foundation named Sunderland as the 15th “most declining” city in the UK, and classified it as an “overshadowed city” because of its proximity to Newcastle upon Tyne. No city from the south featured in the report’s top 24.

Although its population is relatively high – 275,500, according to the last nationwide census in 2011 – its city centre has a considerably lower retail occupancy rate than the nationwide average. Small shops come and go, and some units remain vacant, while the central Bridges shopping centre and out-of-town stores draw much of the trade. The vast department store Joplings has been empty since 2010, the hands of its eggshell-blue clock face stopped at five minutes to one. Empty buildings are a symptom of the economic difficulty Sunderland faces; regeneration is generally considered to be the solution.

The pool is at Newcastle Road swimming baths, taken shortly after its closure in 2009. Photograph: Andy Martin

Martin has explored this abandonment in his work, and witnessed the destruction that invariably precedes regeneration. His night walks took him to old buildings, rooftop car parks, disused railway lines and dry docks.

“I’d get into a building and the bulldozers would be almost there,” he recalls, “or you knew it was going to get redeveloped in a few months, so it was just recording it for posterity, really.”

These photos capture buildings at the very moment their life, their purpose, is slipping away. Budgetary cuts were leaving a trail of abandoned architecture. Martin’s photos make the buildings seem like they’ve been momentarily evacuated; like someone has just stepped out of the room.

A basket filled with children’s swimming floats sits in a drained pool at the vast Crowtree Leisure Centre, while at Newcastle Road swimming baths the pool is still full (Martin got access to the building shortly after its closure). In March 2015, the Gillbridge police station was about to close – its services absorbed by another station outside the city centre – when he got in to photograph its holding cells, the low glow of the CCTV screens in its control centre, and the empty seats of its waiting room. The photos were taken at night, and the lack of natural light reinforces the building’s claustrophobia.

One night, Martin was on his rounds “looking for stuff to photograph” when the Hahnemann Court flat complex, in the Southwick area, caught his attention. “I was blown away by its weird, futuristic architecture,” he says. “There was nothing else like it in Sunderland.”

Martin spent more than five years documenting the closure and demolition of the Hahnemann Court flat complex.

Martin spent more than five years photographing this complex. The chain of four interlinked, six-storey blocks containing 208 flats had been constructed in the mid-1960s from a modular prefabricated concrete system; it sat on stilts above a ground-level car park.

Hahnemann Court had been part of a wave of council housing constructed in Sunderland during a period of slum clearance in the mid-60s; a process of redevelopment that also gave the city a skyline punctuated by high-rise flats. These modern buildings broke dramatically with Sunderland’s dominant architectural form, the terraced house. Hahnemann Court was a concrete megastructure that stood out, especially when surrounded by redbrick streets.

Fifty years later, the complex was earmarked for destruction. When Martin began to take photographs of the building, it was already half empty. “There were whisperings they were going to redevelop it, to modernise the flats and the service areas – but that all evolved into ‘basically it’ll be cheaper to knock it down and build some new accommodation’,” he explains. The residents, some of whom had lived in the building from the beginning, were dispersed to other properties around Sunderland.

By 2011, demolition had begun. Hahnemann Court was lined with asbestos, so “it was a really drawn-out demolition”, according to Martin. “I got pally with the team who were stripping the asbestos out, and they basically gave us free run of the place.”

He was fascinated by the two suspended walkways that linked each building, and eventually climbed on to the roof of one to get a photo. He got into the service room for the building’s lifts, and found messages left by engineers over the years. Martin’s pictures show apparently uninhabited concrete hulks illuminated only by the automated lights in their communal areas. “It took them about 12 months to knock it down,” he says. Rather than new accommodation, an Aldi supermarket was built on the site.

It is impossible to look at Martin’s photographs without wondering about Sunderland’s future. After the EU referendum, worries about the city losing its Nissan factory were assuaged by a secret deal made between the government and the car manufacturer. But questions remain as to how the city, and the north-east region, will fare if a wider economic downturn occurs. The council continues to cut services and, post-Brexit, EU structural funds won’t be available to help with further development.

In September last year, Sunderland launched its bid to become the UK City of Culture 2021. Plans for the revitalisation of the city centre, including the construction of a network of streets, shops and public spaces on the derelict site of the old Vaux brewery, have been unveiled, and a proposal to redevelop the old fire station as an arts centre given the go ahead. According to the bid’s website, the award will “kickstart a four-year period of growth, innovation and creativity”.

“Maybe in a few years, things might have changed,” Martin tells me. “There might be a city of culture. I just feel exhausted with the city as it stands. There’s nothing to challenge us, or to do – creatively, at least. It’s a good time to go.”

Follow Guardian Cities on Twitter and Facebook to join the discussion, and explore our archive here