David Cameron's hopes of striking a deal with the German chancellor to stop Jean-Claude Juncker securing one of Europe's top jobs suffered a significant blow on Thursday when the centre-right and centre-left groupings in the European parliament announced that they wanted a five-year grand coalition behind Juncker as the new head of the EU executive.

In a further setback, the Conservatives were outvoted in Brussels when their own parliamentary grouping agreed to include Germany's new Eurosceptic movement, the Alternative for Germany (AfD), against Cameron's objections and in what Angela Merkel may regard as a hostile act.

The decision to make the AfD's seven MEPs bedfellows of the Tories in Strasbourg suggested that Cameron has lost control of his own creation, the grouping of European Conservatives and Reformists (ECR) he set up when he took the Tories out of the mainstream centre-right European People's Party bloc in 2009.

In a day of frantic political jostling in Brussels three weeks before the new parliament sits, the Speaker, Martin Schulz, a German social democrat, declared that the centre-left, the second-biggest parliamentary caucus, was solidly behind Juncker, the former prime minister of Luxembourg, in his campaign to become next president of the European commission.

Juncker and Schulz led their respective sides in the election three weeks ago – the first ever election to influence the race to head the commission. The process is bitterly opposed by Cameron, who is seeking to muster enough allies among national heads of government to block Juncker.

A head of steam has appeared to be building behind the Luxembourger. Both Schulz and Manfred Weber, the new parliamentary leader of the centre-right and also a German, announced that they wanted to agree a five-year pact, tantamount to a grand coalition that would command a comfortable majority. "Juncker has broad support in the council [of national leaders] and in the parliament," Schulz told the Guardian.

"We are historically in a decisive moment," he said of the attempt to have the voters' verdict determine who heads the EU executive in Brussels. "I want to bring this through. This is my personal ambition. It's a first step. Rome was not built in a day. But this is a major change if we change it … we need to bring the pro-European forces together. I can play a role in that."

Schulz was the architect of the highly contested new system, which saw the election campaign led by candidates for the commission job – a post that has formerly been in the gift of the national leaders of the EU. The result made Juncker the favourite to get the job, but has unleashed a vicious power struggle over who calls the shots in Brussels: the parliament or the national leaders of member states.

Cameron is utterly opposed to Juncker and also strongly against the new system in principle. In an article for a series of European papers, he describes Juncker's nomination as "a power grab through the backdoor", saying that the so-called Spitzenkandidaten system, in which the main political groupings in the European parliament nominate a candidate for commission president, risks politicising the commission.

The prime minister writes: "Many people have deep misgivings about this whole approach … and we should not concede this issue when we know it will set a dangerous precedent for the future."

Cameron uses his article to try to recover some ground after reports that he warned EU leaders that Britain could leave the EU if Juncker was appointed. He says he is not attacking Juncker, described as "an experienced European politician" But the prime minister is struggling to muster allies. A senior parliamentary figure on the centre-right said: "We would like to help the British but we can't solve the problems Cameron has made for himself."

In what would be a nightmare scenario for the Tories, Schulz signalled that he had cut a deal with Juncker to share power in the new commission. "I want to become a member of the commission. Juncker also said he would be happy if the German government suggested Martin Schulz [for the commission]," he said. "Merkel has decided to support Juncker. That is clear. Angela Merkel is a key player, but she is not the only player."

Echoing Juncker's complaints that the British media were engaged in a smear campaign against the Luxembourger, Schulz said: "What the British media are doing with Juncker is a shame. Getting behind his mother-in-law, his family, his house. This is not decent, this is really unfair … to play that kind of game, to try to destroy him … to make him kaputt."

Cameron's stop-Juncker campaign was further troubled when Merkel's anti-euro opponents, the AfD, were admitted to the Tory-led parliamentary grouping because the Conservatives were outvoted.

The Tory leadership moved to distance itself from the decision after a small number of Conservative MEPs supported the admission of the seven AfD parliamentarians in the vote, which was narrowly passed by 29 to 26. Merkel will be annoyed that a group set up by the Tories has given a platform to her opponents.

A Tory spokesman said: "We are very disappointed that AfD has been admitted against our wishes. We note that the vote was a close one. We will work with the AfD in the parliament but [Merkel's Christian democratic] CDU/CSU remains our only sister party in Germany."

The new members mean that the ECR now has 63 MEPs, making it the third-largest group in the parliament, ahead of the liberals. But the admission of the AfD will complicate Cameron's efforts to form a strong working relationship with Merkel in the runup to a renegotiation of Britain's EU membership terms ahead of an in/out referendum in 2017 if the Tories win the general election. The prime minister told Merkel in January that he regarded her CDU party as the Tories' sister party after the Guardian reported that hardline Tories were working to admit the AfD into the ECR group.

The Conservative-led group's new numerical strength is also undermined by the nature of the new recruits, which include the Danish People's party and the Finns party (formerly the True Finns), which are both strongly nationalist, anti-immigrant parties and include a member convicted of racism.

Cameron's Danish, Finnish, German, and Polish allies are more protectionist and nationalist than centrist Conservatives and may resist key Tory policies on Europe, for example, pushing for a big new transatlantic free trade agreement between the US and the EU.