A narrower-than-expected win for Poland’s ruling Law and Justice party (PiS) and a serious setback for Hungary’s governing Fidesz show eastern Europe’s illiberal nationalist parties are not entirely invincible, analysts and commentators have said.

“It looks like this may be a small step in the right direction – but it’s clear the opposition still has an awful lot of work to do,” said Agata Gostyńska-Jakubowska, a senior research fellow at the Centre for European Reform.

Ben Stanley, a political scientist at Warsaw’s SWPS University, said Sunday’s results suggested that when opposition parties “work together and construct a positive narrative, they can convincingly take on politicians who once looked unassailable”.

Cas Mudde, a political scientist and leading populism expert at the University of Georgia, said: “In both Hungary and Poland the opposition seemed to understand the fundamental challenge to liberal democracy they were facing. Strategic collaboration is crucial, particularly when the government party is gaming the system by, for example, controlling the media.”

PiS claimed victory in Sunday’s parliamentary election, winning nearly 44% of the vote according to nearly complete results. The biggest opposition grouping, Civic Coalition (KO), scored 27.2% and the leftist alliance, the Left, 12.5%.

But the ruling party lost control of the senate after opposition parties in most districts united around joint candidates, limiting its control of the legislature for the first time in four years and making it harder for it to push new laws through fast.

It will not enjoy the free rein it has had since 2015 in the lower house either, and could well be forced to seek the backing of MPs from a new far-right group, Confederation, on more controversial policies.

In Hungary, meanwhile, the hardline prime minister, Viktor Orbán, suffered his greatest political setback in a decade when a pro-European, centre-left challenger ousted the Fidesz-backed incumbent as mayor of Budapest by 51% to 44%.

Facebook Twitter Pinterest Viktor Orbán, left, with the outgoing mayor of Budapest, István Tarlós. Photograph: Ferenc Isza/AFP via Getty Images

A wide range of opposition parties from across the political spectrum united to back Gergely Karácsony, 44, over 71-year-old István Tarlós in the capital, with the same tactic meeting success in another 10 of Hungary’s 23 major cities as well.

The mayoral polls were a rare chance for Hungary’s opposition to roll back the power of Fidesz, which holds a supermajority in parliament and had won seven consecutive landslides in national, municipal and European elections since 2010.

The party has used its dominance to reform Hungary’s institutions and concentrate power and much of the national media in Orbán’s hands, and has regularly clashed with Brussels over migration and rule-of-law issues.

The tactic of fielding joint candidates could potentially offer a route to mounting a serious challenge to the strongman prime minister at the next general elections in 2022, although it could prove more difficult to replicate on a national level.

“It proves that the new strategy of opposition cooperation works – this was the opposition’s best result in years,” said one analyst, András Bíró-Nagy. “It is the first crack in the Orban system, and the strategy seems guaranteed to continue.”

But Naz Masraff of Teneo Intelligence said it could prove “hard to form a wide opposition project ahead of parliamentary elections, given the need to align diverging priorities ranging from the far-right Jobbik to Liberal Momentum”.

In Poland, PiS, with little opposition resistance, has also used its absolute majority in both houses to push through deep reforms of the court system that have brought it into conflict with the EU, and turned state TV and radio into propaganda tools.

The ruling party cast the election as a choice between a society of traditional Catholic values and a liberal elite that undermines family life, with critics accusing it of stoking homophobia and anti-LGBT sentiment. It was also rewarded by poorer, mainly rural voters for the 70bn-zloty (£14bn) social and welfare programmes it has rolled out.

“Nonetheless, it was expecting to do better – it said it had not been given ‘what it deserved’,” Stanley said. The party had not been altogether successful in walking the tightrope between keeping its base happy and not alienating floating voters, he said.

“Both the blatant TV propaganda and the anti-LGBT campaign will actually have ended up putting some people off. PiS will not be able to govern in the same way that it has until now,” Stanley said.

But the performance of Poland’s Left alliance in returning to parliament also suggested that “with good organisation and positive messaging, opposition parties can do well” even in environments where they are under pressure, he added.

“This idea that has held sway for so long in Poland and Hungary, that the other side has complete control of the discourse and all we can do is wait for them to make a mistake, looks like it may no longer apply.”

A coordinated opposition with a strong counter-narrative can cut through, agreed Gostyńska-Jakubowska. “Certainly, we can say that some elements of the opposition’s strategy – non-aggression pacts – do seem to have worked. It’s obviously not a uniform picture. But it could be a beginning.”