BERLIN—German voters handed Angela Merkel a disappointing victory on Sunday, forcing her into what will be one of her most delicate balancing acts since she took power 12 years ago.

While the chancellor finished first and is all but guaranteed to head the next government for a fourth term, she scored her party’s worst result since 1949, losing more than one million voters to the right-wing Alternative for Germany, or AfD, according to pollster Infratest-Dimap.

...