For all the horror and the headlines, 82% of Swedish voters failed to cast their ballots for the Sweden Democrats on Sunday, and there is no chance of anti-immigration nationalists taking a formal part in the next government.

But like its Nordic neighbours before it, open, prosperous, liberal, tolerant Sweden is in uncharted waters, facing a right-wing populist insurgency that – despite a performance it will view as disappointing – has now become a very real political force.

Sooner or later – as happened, gradually, in Denmark, Norway and Finland – some kind of accommodation, from ad hoc parliamentary alliance to full-blown coalition government, looks inevitable. It may even be imminent.

Formed in the late 1980s by neo-fascist extremists, the Sweden Democrats have leapt in the space of just three elections from a 5.7% share of the national vote in 2010, when they first entered parliament, to 12.9% in 2014 and 17.7% this year, according to official preliminary results.

The irruption of the radical right party has upended perhaps western Europe’s most stable political order.

For decades, the same two established blocs defined Sweden’s politics, coalescing more recently into a more formal four-party alliance led by the Moderates on the centre-right, and a Social Democrat-Green coalition – backed by the ex-communist Left party – on the centre-left.

The rise of the populists, riding high on a raft of alternative political concerns – immigration, integration, identity, crime, welfare chauvinism – that cut across the traditional left-right divide, is forcing a wholesale realignment.

The Sweden Democrats – until the 2015 migration crisis the only party to consistently and forcefully oppose immigration – have poached voters, leaving the Social Democrats with their worst score in a century, the Moderates well down on 2014, and their respective blocs with no hope of a parliamentary majority.

Facebook Twitter Pinterest Stefan Lofven, the prime minister and leader of the Social Democratic party, with wife Ulla Lofven. Photograph: Jonathan Nackstrand/AFP/Getty Images

They have also exposed deep divisions within the blocs – not least over how best to deal with the Sweden Democrats themselves. They may have rid themselves of most of their overtly racist elements and toned down many of their more extreme policies, but the anti-immigration party remains ostracised, for the time being, by every other.

So where next? No one knows. For the moment, the non-cooperation pact holds. But many analysts – as well as the Social Democrat prime minister, Stefan Lofven – suspect that given the chance, the Moderates in particular may not wait too long to test it. “The cordon sanitaire is starting to fray,” said Nicholas Aylott of Södertörn university.

First, though, Sweden will go through a complicated, messy and potentially long-drawn out process of trying to form a new kind of government – across the traditional left-right divide. That could well prove too much to swallow, especially for the ex-communist Left party.

A German-style “grand coalition” between the main Social Democrats and Moderates is theoretically possible, if unlikely, since the Moderates’ most burning ambition is to oust the Social Democrats. More probable seems another minority government, most likely by the centre-right.

That is feasible because in order to rule, Swedish governments do not need to show they have a majority, merely that they do not have one against them: the new prime minister could end up simply being “the candidate parliamentarians dislike the least,” according to Aylott.

The result would be a potentially vulnerable government, but not necessarily an impotent or an ineffective one: Anders Sannerstadt, a political scientist at Lund university, cites research showing countries used to minority governments often fare better under them, perhaps because they demand pragmatic, cross-party agreements.

But it would, necessarily, depend on the backing of the Sweden Democrats, whose potential to make or break a right-wing minority administration will be considerable. Nonetheless, no one sees Sweden descending into disorder. “We do not expect political chaos or gridlock,” said Ana Andrade of the Economist Intelligence unit. “Sweden has a long history of consensus-based policymaking. The parties will find an agreement.”

Facebook Twitter Pinterest Sweden Democrats party leader Jimmie Akesson speaks during election day in Stockholm. Photograph: Ints Kalnins/Reuters

But if the Sweden Democrats are formally frozen out of government this time, their gradual inclusion in Sweden’s political process – beginning, almost certainly, with occasional parliamentary support for a minority centre-right government – now seems just a matter of time.

“If things carry on as they are going, it will happen,” said Ann-Cathrine Jungar, a far-right specialist at Södertörn university. “In Denmark, Norway, Austria, the far-right is now accepted by the centre-right. I can see that here.”