As you drive through the rich farmland flanking Ringsjön lake in Skåne, the first sign of Hörby is a board proclaiming it “a modern market town with an old heritage”.

The slogan may bring sniggers from big-city Swedes, but this small agricultural town is at the forefront of a movement set to shake the country’s political system this weekend: the emergence of the populist, anti-immigration Sweden Democrats as a serious rival to the traditional parties of left and right.

“My goal is that we should be twice as big as the next party,” said Stefan Borg, the party’s local chairman, sitting in a cafe off Hörby’s main square as he took a break from campaigning for Sunday’s local and national elections.

Borg estimated his chances of becoming mayor by the end of the month at “greater than 50%”. If he succeeds, it would be the first time the Sweden Democrats have led a town council.

The party also hopes to win real influence on a national scale. Since the last general election, support for its policies has surged on the back of unease over immigration. Sweden accepted more asylum seekers per head during the 2015 refugee crisis than any other EU country.

By this June the party had the backing of one in five voters, according to a poll of polls published by Sveriges Radio. Last week the support had slipped back a little, to 17.7%, but even a result like that would represent yet another significant victory for a populist party in Europe, making it difficult for either the centre-right Moderate party or the ruling Social Democrats to form a government without either the populists’ tacit support or deals across the left-right divide.

When the Sweden Democrats became the biggest party in Hörby, with 27% of the vote in 2014, the Moderates refused to go into coalition with them. Partly as a result, Borg expects the Moderates’ vote to sink to single figures on Sunday, while the Sweden Democrats could get more than 35%. “That’s to a large degree because we are the only opposition, and people understand that,” he said.

Facebook Twitter Pinterest Stefan Borg campaigning in Hörby. Photograph: Marita Westerström

It is not just Hörby. The Sweden Democrats hope to become the biggest party in 15 of the 33 municipalities in Skåne. “We’ve always had a historical tradition of being sceptical of anything which comes from Stockholm,” Borg said of Skåne and neighbouring Blekinge. “We are insubordinate.”

At the Boijertz Conditori, a cafe looking over the square to the town’s jugendstil hotel, a woman selling cinnamon rolls summed up the party’s appeal. “It’s because there have been so many problems, a lot of gang fights, knives, drugs being sold openly on the streets,” she said. “It was a sweet little town. You didn’t have to be scared when you went out. And it’s not like that any more.”

One of her customers, Hillevi Gustafson, retorted that this was more perception than reality, pointing to a police survey carried out in December. “It found that more people are feeling unsafe but that nobody had actually had anything happen to them personally,” she said.

“It’s a war zone in Malmö,” Karin, one of three sisters at a neighbouring table, chipped in when she overheard us. “The last two years it’s exploded.”

On Tuesday, yet another young man was shot dead in Malmö, bringing the yearly toll of fatal shootings in the city’s ongoing gang war to 11, or 3.2 per 100,000 people. The shootings have shocked Sweden, and are comparable to London, which has seen 91 murders so far this year, some of them shootings.

Just as they have nationally, the Sweden Democrats here have exploited anxiety over crime. In May, Borg obtained a list of local shelters for asylum seekers and other vulnerable people.

“We have 22 so-called HVB homes here where cities like Malmö and Lund put their problem cases,” he said. “It’s not just immigrants, but the majority are. And of course they come here with all the problems they’ve brought from wherever they come from. Drug abuse is very common.”

Borg, who describes himself as a translator of Russian religious philosophy in the tradition of Dostoevsky, and his colleague Cecilia Bladh In Zito, a former executive at an Italian power company, exemplify the party’s shift away from the working class.

“When I came back here after seven years abroad I was shocked,” In Zito said. “I was like: what has happened?” Even then, it took time for her to become active. “I believed what the TV and newspapers told me, that this was a racist party.”

In 2012 a representative of the party in the town, Ronny Esbjörnsson, was caught railing on Facebook against blatte (immigrant) police officers, saying there should be schools exclusively for “ethnic Swedes” and declaring he would rather rent out a property to Hells Angels than Muslims.

Susanne Meijer, the town’s Social Democrat mayor, accuses the party of still being racist today. She said she was proud that the town welcomed young asylum seekers, whom she described as “talented and smart kids who are studying and learning how to function in Swedish society”.

But she seems to be struggling to convince voters that there’s no reason to be anxious.

On the road out to the north, the council has erected another sign: “Hörby: here we live well and we feel good.”