BERLIN—Germany’s upstart anti-immigrant party beat Chancellor Angela Merkel’s center-right Christian Democrats on Sunday for the first time in a state election, while the center-left Social Democrats held on to win the vote in Mecklenburg-West Pomerania, exit polling and initial results showed.

The three-year-old Alternative for Germany, or AfD, was set to receive 21.4% of the vote, projections from pollster Infratest Dimap said after the polls closed in the northeastern German state. The Christian Democrats, by contrast,...