Swedes headed to the polls Sunday in an election in which an anti-immigration party rooted in neo-Nazism is threatening to upend the political order in one of Europe’s most traditionally liberal nations.

Following a polarizing campaign dominated by the issues of immigration and crime, the nationalist Sweden Democrats are poised to win between 17% and 24% of the vote, according to opinion polls, up from 12.9% in 2014.

Any...