Michel Barnier has described Boris Johnson’s solution for replacing the Irish backstop as “unacceptable” as the EU’s chief negotiator gave his most downbeat assessment yet of the chances of striking a Brexit deal by 31 October.

Standing alongside the German foreign minister, Heiko Maas following a meeting in Berlin, Barnier said the UK government’s position had to change for there to be any hope of success.

The British prime minister had declared on his way to New York for the UN general assembly that he was “cautiously optimistic” he could persuade key EU leaders to accept his proposals for the Irish border.

Johnson wants to find agreement on a deal ahead of an EU summit on 17 October that he can put to parliament in order for the UK to leave the bloc at the end of the month.

But as he sought to break the Brexit impasse by appealing to EU capitals and the European council’s president, Donald Tusk, Barnier appeared to scotch any hope of a rethink in Brussels.

The EU insists it will not sign up to a deal that creates a regulatory and customs border on the island of Ireland, as proposed by the UK in the papers tabled last week by the government’s chief negotiator, David Frost.

Barnier said: “The new government of the UK wants us to get rid of this solution, the so-called backstop and wants … a regulatory and customs land border on the island of Ireland.

“The UK government also wants the EU to change the way the internal market and border control operates after Brexit.

“As I am sure you will understand, this is unacceptable. My mandate is clear of the 27 leaders, the EU and the European parliament, safeguarding peace and stability in Ireland and protecting the integrity of the single market.

“Let me therefore put it clearly that based on current UK thinking, it is difficult to see how we arrive at a legally operable solution that fulfils all the objectives of the backstop.”

Barnier went on to insist the EU was open to “new ideas” should they be proposed. But he added that the talks were in a “very difficult sensitive phase”.

“The ball is in the court of the British,” he added.

The UK government has proposed maintaining an all-Ireland sanitary and phytosanitary zone for agrifood products to ensure that the flow of 30% by value of the trade across the border is unimpeded.

The EU has said this “partial” solution does not avoid additional customs and VAT checks on the island of Ireland nor deal with the issue of manufactured goods that cross the border after Brexit.

The backstop in the withdrawal agreement would in effect keep Northern Ireland in the single market and the whole of the UK in a shared customs territory in order to maintain peace, protect the north-south economy and avoid border infrastructure of even additional related checks.

Of the UK’s suggestion that customs paperwork could be done away from the border and that technology and trusted trade schemes could facilitate the necessary checks, Barnier and Maas raised doubts.

“Objectively, there are possibilities,” Barnier said, but added: “I don’t know how to inspect a cow with virtual methods.”

Q&A What is a Northern Ireland-only backstop? Show The British government’s version of Brexit involves the UK ultimately leaving the single market and customs union, requiring the return of a range of checks on goods crossing the border between Northern Ireland and the Republic of Ireland. The “backstop” is intended as a standstill placeholder to ensure such checks do not have to be imposed between Brexit happening with a deal, and the start of a new free trade agreement yet to be negotiated between the UK and the EU. Theresa May's withdrawal agreement proposed keeping the whole of the UK in a shared customs territory with the EU during this period. An alternative idea involves only Northern Ireland staying in the EU’s customs territory. That would place a customs border in the Irish Sea. May described it as a threat to the constitutional integrity of the UK, but the new prime minister, Boris Johnson, has opened the current talks by proposing an all-Ireland agri-food zone. The suggestion is that he will seek to quietly build on that with further NI-only arrangements. Given an NI-only backstop was an EU proposal in the first place, the U-turn would be warmly welcomed in Brussels, although attempts to give the Northern Ireland assembly a veto on its continuation would not be acceptable, and the DUP would be unlikely to support the prime minister in such a move in parliament. If there is a no-deal Brexit, then there is no backstop. Daniel Boffey

Maas said it was “progress” that the UK had at least made some proposals, but that “even from a British point of view these presented ideas are not a legal and viable solution”.

He added: “I have never believed that digitalisation can solve all problems … In the future what we will do is create a European position to this in close consultation with Michel Barnier. At this moment, all we can say is just that the suggestions are not legal or viable solutions as they have been agreed on in the exit agreement. But this would have to be the case.”

Johnson is meeting the French president, Emmanuel Macron, the German chancellor, Angela Merkel, and the Irish taoiseach, Leo Varadkar, along with Tusk while in the US.