The anti-immigration Alternative für Deutschland party made strong gains in two crucial state elections in Germany on Sunday, increasing its support significantly but failing to oust the mainstream parties.

But the sharp shift to the right in Saxony and Brandenburg – AfD came second in both states – is a blow to the ruling coalition of Angela Merkel’s Christian Democrats (CDU) and the Social Democrats (SPD), both parties having lost thousands of voters to AfD.

The AfD was also able to mobilise several hundred thousand people who had never voted before, initial analysis showed.

Exit polls showed the CDU remaining the strongest party in Saxony but losing more than six points to secure 33%, while AfD reached 28.1% – a gain of 18 percentage points, and a larger share of the vote than pollsters had predicted.

In Brandenburg, the SPD, which has governed there since 1990, narrowly clung to first place, winning 26.6% and losing 5 points, while AfD secured 24.5%, a more than 10-point rise and a larger share than predicted.

AfD’s success in Saxony and Brandenburg, both in the former communist east, reflects the breakdown of support for Germany’s mainstream parties, the centre-right CDU and the left-of-centre SPD and, as elsewhere in Europe, the increasing fragmentation of the political landscape.

More than 5 million people were eligible to vote, about a 10th of the population.

Turnout was significantly higher than at the last elections in the states in 2014 – up 12 points to 60% in Brandenburg, and up 16 points to 65% in Saxony – in what was billed a historic poll, AfD’s first real electoral test in the region since it entered the national parliament as the leading opposition party two years ago.

Formed in 2013 as an anti-euro party, its strength has grown on the back of its opposition to the arrival in Germany of almost 1 million refugees in 2015.

It campaigned in Saxony and Brandenburg under the slogan “Vollende die Wende”, or “complete the transition” – promising to rectify the mistakes of the mainstream parties after German reunification almost three decades ago and to address the inequalities between citizens of the former east and west.

Björn Höcke, a leading AfD member based in the state of Thuringia, which votes on 27 October in what is expected to be another nail-biting election, said the polls offered “a strong indication that we’ll have a good future in Brandenburg and Saxony and in the whole of Germany”. Höcke represents AfD’s radical right contingent, whose influence in the party is likely to be strengthened by the result.

The Green party was celebrating gains in both states, of 3.3 points in Saxony, where it reached 8.6%, and 3.8 points in Brandenburg, reaching 10.2%, even though it failed to perform as strongly as had been predicted.

But it is likely to be an important player in the formation of coalitions in both Saxony and Brandenburg, where the SPD-Left party partnership lost its majority. All the major parties have ruled out forming a coalition with AfD, even though the CDU has come under pressure from its right wing to do so.

Because the slump in the performance of the mainstream parties was not as severe as predicted, the beleaguered grand coalition in Berlin under Merkel will gain some breathing space, as will the CDU’s chairwoman, Annegret Kramp-Karrenbauer.

She has failed to convince as a leader since taking over from Merkel in December, but will probably benefit from the CDU’s better than expected showing in Saxony. Merkel, who has led the country since 2005, has said she will not stand for re-election. The next German federal election is expected to be held between August and October 2021.

Analysis of the results showed that in Saxony, home to the notorious anti-immigrant campaign group Pegida, often seen as the protest arm of AfD, voters’ main motivation for choosing AfD was out of conviction for its political message, while the majority of AfD supporters in Brandenburg said they voted in protest against the policies of the mainstream government.