It started, as many days do in Greece, with a trip to the kiosk to buy cigarettes. Still half-asleep, Panayiotis Roumeliotis was surprised to be asked to show his identity card by two young men with shaved heads. It was his first direct contact with the vigilante groups that have become a feature of everyday life in some areas of the Greek capital.

"They were calling themselves the residents association but they were just fasistakia (little fascists)," said the 28-year-old.

Over the last two years, Mr Roumeliotis has watched the central Athens neighbourhood of Ayios Panteleimonas, where he grew up, undergo an ugly transformation. Taking the bus on another morning soon after, a gunshot shattered the back window and a gang of men forced the driver to stop. When the doors opened, they came on to the bus and started to assault the non-Greek passengers. The attackers were wearing T-shirts from the right-wing extremist group Golden Dawn. While panicked people were trying to escape from the bus the men were hitting them with flagpoles.

"They were beating people with the Greek flag," said Mr Roumeliotis.

When the police arrived they stood off until the thugs had finished. When he asked the police why no one had been arrested one of the officers replied to him: "Why, did they do something to you?"

Formerly a solid middle-class neighbourhood, the economic crisis and waves of new arrivals have changed the area and erased old certainties.

Property prices here have dropped to as little as one quarter of what they were five years ago. The Greeks who could afford to have left. For rent signs are plastered over almost every one of the area's shabby five-storey apartment blocks. On the side streets among the North African-run mini markets and Nigerian internet cafes, newcomers from West Africa push shopping trolleys full of scrap metal stripped from deserted buildings. Large-scale drug dealing has overtaken an entire street in the neighbourhood. Violent crime has rocketed.

The square in front of the local church, daubed in anti-immigrant slogans such as "foreigners don't fit in our square", has witnessed pitch battles between anarchists and Golden Dawn supporters.

Among the Greeks who remain, Mr Roumeliotis' own circumstances are fairly typical. He was made redundant from the "job for life" that his grandfather had got for him at the state-run airline Olympic Airways. Of his 10 closest friends, eight are unemployed.

Anger against the socialists and the conservatives, who have swapped power since the end of the dictatorship in 1974, has been building to unprecedented proportions. The former airline steward said he would not be supporting the fascists himself, but that other family members may well be doing so.

"People here have been forgotten by the government," said Mr Roumeliotis. "They have done nothing about immigration." The skinheads are now talked of as "good boys" who are looking out for their community. What was happening in Ayios Panteleimonas – which has the highest concentration of immigrants in Athens – came to the attention of the rest of Greece when members of Golden Dawn were voted on to its local council.

Until recently, developments in the neighbourhood were seen as dangerous but largely irrelevant to the national scene. At Greece's last general election in 2009 Golden Dawn, whose members use the Nazi salute and whose party symbol is an adapted swastika, polled fewer than 20,000 votes nationwide. Now as the country goes to the polls on Sunday, national politics more closely resemble those of the embattled area.

Entering its fourth year in recession, Greece now outstrips even Spain for youth unemployment with the new statistics published yesterday showing joblessness among the under-25s at 51.2 per cent. The headline unemployment rate is 21.7 per cent while the real rate is believed to be closer to 25 per cent.

It's in this environment that a fringe group of neo-Nazis that would previously have struggled to attract a hundred supporters to one of its rallies seems set to enter parliament according to polls.

Together with two other ultra-nationalist parties, LAOS and Independent Greeks, some surveys indicate that the far right could take as much as one-fifth of the vote on Sunday.

Under Greek law no surveys of voter intentions can be published in the fortnight before the election but unpublished polls conducted by the major political parties have been giving Golden Dawn a steady five per cent, with one poll predicting eight per cent. The threshold for entering the 300-seat parliament is three per cent and under Greece's convoluted electoral laws that could give them somewhere between eight to 12 MPs.

One of the new lawmakers is likely to be Elias Panayiotaros. A candidate for central Athens, he runs a shop called Phalanx selling military gear. His business card features a Byzantine eagle and a skull and crossbones. Speaking at the party's offices opposite the Larrisis train station where volunteers do weekly handouts of clothes and food to "needy Greeks", Mr Panayiotaros said that the priority in the midst of the economic crisis was to throw out foreigners. "All of the immigrants are illegal, even the ones that have been in the country for a long time and they have to be punished," he said.

Greeks were afraid to leave their homes as immigrants were on a spree of robberies, rapes and murder, he said. The cost of jailing or deporting more than a million people would be offset, he said, by putting them into work camps. Those that were not prepared to work would not be given food, he added.

Another priority would be the immediate closure of all foreign non-governmental organisations, which would be compelled to return all funds to the Greek state. A new organisation would be established, he said, to be called "doctors with borders" which would provide medical services exclusively to Greeks.

Once Greece was withdrawn from the EU's passport-free Schengen zone and the borders were sealed, all foreigners would require a visa to visit Greece, including the millions of tourists who come each year.

"We're not afraid to be called Nazis or fascists – the whole political system is against us," said Mr Panayiotaros. Appeals by mainstream parties for Golden Dawn to be prevented from entering parliament were met with veiled threats against MPs.

Asked about the party leader Nikos Michaloliakos' links to the military junta which ruled Greece between 1967-1974, he said that people were calling for the return of George Papadopoulos, the leader of the Colonel's regime.

"The name of Papadopoulos is being heard everywhere," he said.