The longest ever EU summit has ended without agreement as Angela Merkel warned that with Brexit “looming” imposing the centre-left candidate Frans Timmermans as European commission president risked creating a dangerous split with the populist governments in Poland and Italy.

With the leaders now forced to meet again in Brussels on Tuesday after being unable to agree on a candidate for the top post, the German chancellor said fears about the bloc splintering left her wary of trying to outvote critics of her compromise plan.

The so-called Visegrád countries – Poland, the Czech Republic, Slovakia and Hungary – had used the marathon 20-hour summit to rail against Merkel’s plan to replace Jean-Claude Juncker with his current deputy in the commission, who has been stern critic of governments in eastern Europe who have threatened their country’s judicial independence.

Q&A What is the Spitzenkandidaten process? Show The Spitzenkandidaten process is a method by which the European Parliament seeks to influence who is appointed as President of the European Commission - a role held from 2014 to 2019 by Jean-Claude Juncker. Under the process, each political grouping in the European Parliament proposes one 'lead candidate' to take the role. The presidency subsequently goes to the political party winning the most seats, or the one that brings together the widest coalition of support for their candidate. However, this is not a formal process set out in any treaty or constitution. The candidate for the role is nominated by the European Council, made up of the leaders of the EU member states. They are able to ignore the Spitzenkandidaten nomination and to chose someone else. The appointment has to be subsequently ratified by the European Parliament though, and therefore choosing an alternative candidate risks provoking conflict between the institutions.

The Italian prime minister, Giuseppe Conte, who leads a coalition government of the far-right League party and the M5S, lent his support in efforts overnight to scupper Timmermans’ candidature, which had been agreed between Merkel and the French president, Emmanuel Macron, at the G20 summit in Osaka before the meeting of the EU’s 28 leaders.

Emerging from the record-breaking talks, Merkel told reporters: “We have to pay attention to the fact that we have smaller and bigger countries in the Visegrád group but to outvote that group and, on top of that, a country like Italy that would be difficult…

“It should not lead to tensions that will determine years and years to come. The Brexit is looming on the horizon. Other important issues are on the table. I think we need to treat each other with care.”

Timmermans, 58, an Anglophile and critic of the “borderline racist” past comments of the Tory leadership frontrunner Boris Johnson, had been proposed by Merkel as the compromise candidate who could command a majority in the European parliament.

Parliament has a veto on anyone proposed by the 28 heads of state or government and has insisted that it will only support one of the so-called Spitzenkandidaten, the parties’ lead candidates, who campaigned during the elections to be commission president.

The German MEP, Manfred Weber, the lead candidate of the centre-right European People’s party, which emerged as the biggest group, is unacceptable to Macron due in part to his lack of government experience.

Timmermans, a former Dutch foreign minister who speaks seven languages, was the Spitzenkandidat of the Socialists and Democrats group, which came second in the elections.

Under the rules, at least 21 heads of government representing 65% of the EU’s population need to back a candidate for him or her to go forward to be put to a parliamentary vote.

But Merkel said that while that number would be sufficient it would not be “satisfactory”. “What is important for me is that we don’t end with 65.1% of the majority of the population,” she said. “That would be a bit meagre I believe.”

She continued: “We have major member states who found the proposal unacceptable and when you have 440 million inhabitants, [you ask] whether you just want to outvote 100m – formally it would be possible – or whether you invest more time and try to strike a compromise and find a common path together.”

Merkel’s proposal was also challenged by fellow members of the centre-right EPP, including the Irish prime minister, Leo Varadkar, who argued against giving up the top job to the centre-left.

Macron emerged from the summit to describe it as an embarrassment.

The visibly angry French president said: “We ended the day on what we can call a failure. It’s a very bad image we are giving of the council and Europe, no one can be satisfied with what happened over so many hours.

“This failure is that of division, political division in the EPP that marked the dissension of many of its leaders and geographic division within the council that led to deciding to reconvening tomorrow, and to find the necessary rest and for calm to return and a new solution that requires some to move a little.

“Our credibility is profoundly tainted with these meetings that are too long and lead to nothing, we give an image of Europe that isn’t serious.”

“Everything went wrong and obviously the result is very frustrating,” said the Portuguese prime minister, António Costa, with reference to “the council’s incapacity to take decisions and construct solutions that have majority support in the council and European parliament.”

Costa said some leaders had been “deeply engaged” to find a solution but others were “captured by those who want to divide Europe, from the Visegrád group or from positions such as Mr Salvini’s [Italy’s far-right deputy prime minister].”

Merkel said there would be a time-limit on the leaders’ deliberations as the European parliament would appoint its president on Wednesday, and the European council wanted its nomination to be made before then.

The leaders also need to select a replacement for Donald Tusk as European council president, Mario Draghi as president of the European central bank and Federica Mogherini as high representative of foreign affairs.