“We don’t wish to be sounding nimby, or that we’re not keeping up with the times,” worries Will Windell, a Southwold councillor, “but you need to see the effect that this stuff has in a real way on a small community.” The stuff, for the full-time residents of Suffolk’s picturesque coastal town, is the news that second-home owners in the area are exploiting a legal loophole to dodge council tax. “We’re subsidising all the local services,” says Windell, exasperated, in an impromptu meeting conducted over a dining table. “We pay for the refuse collection and the roads; these guys don’t, and it’s distorting the local economy.”

Earlier this month David Beavan, a retired farmer and Liberal Democrat candidate in the upcoming Southwold and Reydon byelection for the district council, researched Southwold’s most desirable streets – rows of grand townhouses and charming cottages closest to the beach – and discovered that 260 properties were registered as small businesses, intended as holiday lets for a minimum of 140 days per year and without any obligation to fulfil that requirement or to then pay any council tax. According to his follow-up survey, around 80% of those houses stood empty most of the time.

“There are about 1,400 houses in the town,” says Beavan, over a pint of cider outside the Red Lion, “and roughly 900 of them are second homes and 500 are lived in full-time.” The tension between locals “and the London media set that come some weekends” has simmered for more than two decades, with the latter inflating property prices and squeezing the former out, but the real upset now is reserved for what he calls “the new breed of property investors”.

“Traditional second-home owners come here and holiday here with their children and their friends, they maybe let their homes out a bit, they retire here, they contribute to the town and pay their taxes, they’re fine,” he says. “The ones we’re against are the ones who buy up property, register it as a business so they don’t pay anything, don’t visit or contribute anything to the town, leave it empty and flog it in five years’ time for profit.”

The matter came to national attention on Wednesday when it was raised in the House of Lords by Lord Shipley, the Liberal Democrat peer, and Lord Deben, the former Conservative MP for Suffolk Coastal John Gummer, who are both lobbying for legislative change.

“The council could collect around £300,000 a year if these homes were properly taxed,” estimates Beavan. “But this government won’t take it from these rich people – it’s probably their friends.”

Inside the pub, unprompted, the barmaid chats to a friend about the “astronomical prices” of Southwold’s housing stock – around half a million pounds for “a small two-up, two-down” property, in a town where the average salary is £19,459 a year and local brewery Adnams is the biggest employer.

Lifeguard Simon Callaghan says that “Southwold has become God’s waiting room”. The population of 800 or so is dominated by the elderly. Locals complain that young families are forced to move to the nearby village of Reydon or neighbouring towns because there is little housing to rent or affordably buy.

Jessica de Grazia, a former barrister who has lived on and off in Southwold since 1987, is fed up with the town’s skewed economic reality. “This area has a crying need for rental housing for people to live in year round, we have so very little of that. It’s a minimum-wage economy, it’s shameful that young people can’t afford to live here.”

Inside her detached corner house, where the aforementioned dining table committee has gathered, she is energised. “The reason we’re getting so many holiday lets is the government and the taxation rules,” she says. “All the government has to do is change the financial incentives, but we need support and people jumping on the the bandwagon, because I’m not sure government even realises this loophole exists. If they knew, surely they would close it – it’s not in anyone’s interests to excuse holiday lets from paying rates.” De Grazia’s home is one of the few on her road lived in full-time; it’s where she runs SouthGen, a community group that has campaigned with a housing association to buy Southwold’s decommissioned hospital from the NHS for the community.

“We got it for about 40% of what they could have sold it for to developers,” she says proudly, “and we’re raising money to develop it into affordable housing, community and business space for start-ups, a creche, a library and a farm-to-fork cafe”.

In cheery sunshine across the peacefully posh part of town, Beavan conducts a glum tour, pointing out the spots where only one or two permanent residents live on an entire street. For 15 minutes, he simply mutters: “That’s a holiday let … that’s a holiday let … that’s a holiday let”.

“Everywhere – apart from Wales where they changed this – a lot of people are just banging their heads against bricks walls,” he says. “[MP] Andrew George from St Ives was saying this five years ago. Everyone nods and says that needs to be sorted out and nothing happens.”

He’s been accused of “whingeing on” about the subject by second-home owners on Facebook, but it doesn’t bother him. “The issue has hit a crux now: if it goes beyond 65% here with holiday homes, people won’t want to come here. It’ll just become a second-home village, and we’ll have killed the goose that laid the golden egg.”

• This article was amended on 27 June 2018 to clarify that Jessica de Grazia has lived on and off in Southwold since 1987.

