BERLIN—German voters dealt a stinging rebuke to Chancellor Angela Merkel and her open-door refugee policy in three state elections Sunday, delivering historic gains for an upstart anti-immigrant party and showing how the migration crisis is scrambling politics in Europe’s largest economy.

The populist Alternative for Germany, which focused its campaign on opposition to Ms. Merkel’s migrant policy, won nearly a quarter of the vote in the eastern state of Saxony-Anhalt. The result—several percentage points higher than recent polls had suggested—represents the party’s best total in a regional election since its founding three years ago.

The party, known as the AfD, also won parliamentary seats in two former West German states, giving it representation in eight of the country’s 16 state legislatures. That strengthens the AfD’s status as a significant political force to the right of Ms. Merkel’s conservative bloc—a turning point that her Christian Democrats long tried to prevent.

At the AfD’s election-night celebration in Berlin, supporters chanted “Merkel must go!” after the results started to come in.

“We have a very clear position on refugee policy: We don’t want to take in any refugees,” AfD deputy chairman Alexander Gauland said on German television. “All who voted for us stand behind this policy.”