Angela Merkel has spoken of growing difficulties in striking a Brexit deal as the European council president, Donald Tusk, warned that a no-deal scenario was “more likely than ever before”, following the latest derailment of talks over the Irish border.

“We were actually pretty hopeful that we would manage to seal an exit agreement” but “at the moment, it looks a bit more difficult again”, the German chancellor told the German Foreign Trade Federation after the latest talks, which have largely been held in secret.

A breakthrough was still possible, Merkel added, but it would need “quite a bit of finesse and if we aren’t successful this week, we’ll just have to keep negotiating”.

Tusk, in his invitation letter to a leaders’ summit, starting on Wednesday evening with a dinner to which Theresa May will present her thoughts on the Brexit talks, said the recent negotiations had “proven to be more complicated than some may have expected”.

He said: “We should nevertheless remain hopeful and determined, as there is good will to continue these talks on both sides. But at the same time, responsible as we are, we must prepare the EU for a no-deal scenario, which is more likely than ever before. Like the UK, the commission has started such preparations, and will give us an update during the meeting.

“But let me be absolutely clear. The fact that we are preparing for a no-deal scenario must not, under any circumstances, lead us away from making every effort to reach the best agreement possible, for all sides.

“This is what our state of mind should be at this stage. As someone rightly said: ‘It always seems impossible until it’s done.’ Let us not give up.”

On Sunday, the Brexit secretary, Dominic Raab, made an unscheduled visit to Brussels to inform the EU’s chief negotiator, Michel Barnier, that the UK government was not willing to sign up to a “backstop” solution for avoiding a hard border on the island.

The prime minister told the House of Commons in a statement on Monday that the EU was insisting that its backstop, in which Northern Ireland could in effect stay in the single market and the customs union as the rest of the UK withdrew, remain in the withdrawal agreement. Theresa May said that this would be neither legal under British law, nor feasible for any prime minister to accept.

Despite the tough comments, EU officials insisted that the negotiations had been merely been put on “pause” rather than having suffered a breakdown.

There remains a large degree of agreement between negotiators on both sides, and texts are being worked upon, with diplomats in Brussels believing there will be a moment where the prime minister will be able to risk cabinet resignations and the wrath of Northern Ireland’s Democratic Unionist party to strike a deal.

The cautiously optimistic tone of officials was further echoed by the French president, Emmanuel Macron, who had demanded “maximum progress” by this week’s leaders’ summit to allow an extraordinary Brexit summit to be called in mid-November.

“I believe in our collective intelligence, so I think we can make progress,” Macron said.

A senior EU diplomat said the leaders would decide on Wednesday evening whether a November summit was feasible given the lack of obvious progress. “May in the House of Commons didn’t sound too pessimistic and I think we’re close but we still need some time,” the diplomat said. “We are close probably on the withdrawal agreement, but on the future relationship there are still some things open, so we probably need more time to do that.

“Whether a summit happens in November depends on the dynamics of Wednesday evening and there are some heads of state that are rather reluctant while others are more forthcoming so we need to see that.”

May will not be joining the 27 EU leaders for their dinner after she has made her comments. “I am sure she will be given something [to eat]”, a diplomat said. “We are mean but not that mean.”

Officials did not, however, offer May any hope that it would dispense with its belief that the withdrawal agreement would need to maintain a version of the backstop that keeps Northern Ireland in the customs union and single market, should other plans, including an EU-UK customs union, fail to materialise.

”I don’t see anything acceptable that leaves us with a cliff-edge at a certain point in time,” a diplomat said. “We are working on the assumption that we want to avoid it and we have the political will. We are still positive we can achieve it, so there’s no reason to despair.”

Ireland’s foreign minister, Simon Coveney, said the failure to reach a deal on Sunday was “frustrating and disappointing from an Irish perspective”, adding that an agreement was “clearly going to take a bit more time than many people hoped”.

At a meeting of EU foreign ministers in Luxembourg, he called on the British government to follow through on commitments made in December and March. The British government promised in December to maintain “full alignment” with the EU single market and customs union to protect the Good Friday agreement, in the absence of a trade deal. Then, in March, the prime minister reiterated that she supported “an operational legal text” for the backstop.

Brexit talks hinge on how those promises are interpreted. The British government is seeking to avoid an open-ended Irish backstop, but the EU says a time limit makes the guarantee worthless. “Nobody was suggesting in March that the backstop would be time limited,” Coveney said.

Arriving at the meeting in Luxembourg, the British foreign secretary, Jeremy Hunt, put a brave face on the setback.

“It was always going to be a moment like this,” he said adding that “a huge amount of progress” had been made. “There are one or two very difficult outstanding issues, but I think we can get there. Whether we do it this week who knows, but I know everyone is trying incredibly hard.”