The Greek prime minister, Antonis Samaras, has been struggling to contain a fresh political crisis after his chief of staff was filmed accusing him of instigating a judicial inquiry against the neo-Nazi Golden Dawn party for political gain.

The gaffe led to the resignation of Takis Baltakos, Samaras's most trusted adviser, with the prime minister facing mounting calls from across the political spectrum for the government to step down.

"The prime minister cannot go on hiding," railed Alexis Tsipras, leader of the radical left main opposition party, Syriza. "The Samaras government has proved to be dangerous; it cannot continue handling the country's fate."

The aide's resignation on Wednesday came hours after the release of a video showing a highly embarrassing exchange between Baltakos and Golden Dawn's spokesman, Ilias Kasidiaris. During the meeting, secretly filmed by Kasidiaris, the cabinet secretary is heard saying that a criminal investigation into the group was aimed purely at stemming losses for the ruling conservative party. "He's afraid for himself because you are cutting his lead over Syriza," he tells Kasidiaris in the video posted on YouTube.

In a febrile political atmosphere ahead of crucial European parliament elections, the recording has reinforced the perception of links between the far-right and factions in Samaras's New Democracy party.

Greece's third-biggest political force until the crackdown, Golden Dawn has long argued that the inquiry is politically motivated. Since the fatal stabbing of a leftwing rapper by a party cadre last September, six of its MPs, including Nikos Michaloliakos, who founded the group more than 30 years ago, have been imprisoned pending trial on charges of operating a criminal organisation.

This week parliament voted to lift the immunity from prosecution of another five deputies, bringing the number of politicians who face allegations of orchestrating attacks against immigrants, political opponents and members of Greece's gay community to 13. "They must be immediately set free," said Golden Dawn's parliamentary representative, Ilias Panayiotaros, repeating claims that the judiciary was being exploited by a rotten political establishment.

Baltakos, who co-ordinated government policies as Samaras's chief of staff, attempted on Thursday to distance the leader from what he described as a personal mission to stop Greeks from voting for the extremists. Golden Dawn's meteoric rise has been attributed to a shift in support by traditional rightwingers disgusted with austerity measures.

"The prime minister did not know. I was operating on my own initiative as a mediator for the good of the country," said the former aide, insisting his aim had been solely to "dislodge" the thousands of Greeks who had voted for Golden Dawn.

"The 500,000 Greeks who voted for Golden Dawn are not Nazis … and for the good of the country someone had to meet with these people," he told a local radio station, adding that "very possibly" other secretly recorded meetings would also come to light. "I told them what they wanted to hear so I could win their trust."

But with only weeks before European and local elections – and Samaras' parliamentary majority whittled down to one following an exodus of support over unpopular budget reforms – many predict a loss of votes for his fragile two-party coalition.

Pasok, the government's centre-left junior partner, issued a statement demanding that the conservatives' ties "with neo-Nazis, racists and the extreme right be revealed and punished".