It has been described as the worst place to live in Britain, its locals known as "smoggies" in a nod to its association with heavy industry. But now Middlesbrough is fighting back, and when the council waved through plans to turn the town's only Grade I-listed building into a wedding venue with 56 "executive homes" and a "health village" in its grounds, it proved the final straw.

A group of enraged residents, who say they were only consulted once about the plans, have started a campaign aimed at protecting the future of Middlesbrough, which they say is increasingly becoming known for "boarded up houses, alcohol, drug addiction and crime".

The group, Hands on Middlesbrough, said they were sick of their town being described as the "next Detroit" and wanted to "protect our heritage and green space before the Blade Runner legacy becomes our reality". Nowhere, they claim, is this erosion more evident than at Acklam Hall, built in the 1680s and Middlesbrough's oldest domestic building. English Heritage has described it as "of exceptional national interest" and "one of the North-east's architectural jewels".

In April, the body's principal inspector of historic buildings, Catherine Dewar, wrote to Middlesbrough Council describing the plans as "extremely disappointing" – but the following month the council unanimously approved them, saying it was the only way to raise the £2m to £3m needed to restore the Hall, which has fallen into disrepair.

Another critic of the development is Jonathan Foyle, the architectural historian and broadcaster, who described it as "banal in the extreme" and "the final waste of a fine historic asset". He accused the council of employing "the old tactic of underinvestment in historic places and then claiming that redevelopment is essential".

Demolition (Alison Brown) (Getty)

Hands on Middlesbrough says the council did not consider alternative sources of cash such as lottery funding before agreeing to the plans, which also required the chopping down of 200 trees with preservation orders attached to them. The town's mayor, Ray Mallon, described building more homes on the site as "the only game in town".

Scarlet McBride, 33, a theatre worker, is one of the leaders of the new group. She says its aim is to encourage local people – especially the young – to reject the idea their town is a lost cause and to start a debate about the sort of place in which they would like to live. "I think if we continue to disrespect our heritage and our green space, then it will become our reality," she said. "There'll be no historical significance to the place, nothing of character that sets us apart from any other housing estate, as our land becomes homogenous.

"With Acklam Hall being Middlesbrough's only Grade I-listed building, it means that a lot of other places will be under threat, because if they can build there they can pretty much build anywhere. You wouldn't build houses on Downton Abbey – it doesn't seem right. They need to look at alternatives first, rather than just deciding that the only way is development. The public have been shocked by what has happened."

A stone rose (Alison Brown) (Getty)

Once a giant of steel and iron production with the nickname "Ironopolis", Middlesbrough is now said to be part of Britain's "rust belt" – a band of former industrial towns and cities deemed in terminal decline. Some of statistics make grim reading: between 2011 and 2012, for instance, child poverty rose by 38 per cent. Unkind jokes are common, such as: "What's the best thing to come out of Middlesbrough? The A66."

In 2007, the property programme Location, Location, Location labelled Middlesbrough the "worst place to live" in the UK. Presenter Phil Spencer said: "Ninety per cent of residents never exercise and few eat healthily… robbery, burglary, sexual assault, violent crimes and car theft are all more than twice the UK average." The show prompted numerous complaints – which were not upheld.