It was an unseasonably warm and sunny October day when the idea of the “tunnel”, the secretive negotiating phase from which Wednesday’s deal has emerged, was first raised with EU ambassadors in Brussels.

The Salzburg summit had just passed with the limits of the negotiations hence far having been cruelly exposed.

The prime minister had felt humiliated by the EU’s rejection of her Chequers proposals while the UK was perceived by other European countries to have attempted to go over the head of Michel Barnier, the EU’s chief negotiator, by both accusing him of stubborn gallic inflexibility and appealing to the leaders to intervene.

That diplomatic misstep by Downing Street had merely strengthened the French official in the eyes of the EU. But it certainly soured relations with the British. “They won’t be happy until Barnier’s publicly flogged,” one senior EU diplomat in the Austrian city said at the time.

The European commission decided to use the moment.

At a meeting in the European council’s Europa building on Friday 5 October, the EU’s German deputy chief negotiator, Sabine Weyand, regarded as the brains of the EU’s negotiating team, sought to cash in on Barnier’s popularity with the member states.

Barnier had delivered on the £39bn divorce bill, and on citizens’ rights earlier in the year. The talks were getting bogged down on the all-weather or backstop solution for avoiding a hard border on the island of Ireland, she said. The UK could not accept a plan that could keep Northern Ireland in the customs union while Britain withdrew at the end of the 21-month transition period.

“We need to give it a chance,” Weyand, a Cambridge-educated student of Shakespeare, said of the talks.

Weyand told the room of diplomats that she wanted to take the negotiations out of the public eye, and talk intensely with the British, without the usual consultation with the member states laid down in a hefty book of rules of procedures.

The small core group of negotiating teams would set up home on the fifth floor of the European council’s Berlaymont building to throw around new ideas, some of which were potentially highly controversial, to try to break the deadlock.

Sabine Weyand: ‘If we fail to plan for success, we plan to fail.’ Photograph: European commission

“What we need is for you to trust us,” Weyand told them. “We don’t know where we will land but it is worth exploring, we don’t know how it will work but trust us.”

“If we fail to plan for success, we plan to fail,” she added. “What we would really like to do is go into a tunnel and then come back to brief you about what happened just before the next European council.”

The ambassadors readily agreed.

The logic of the tunnel had been to play with ideas in a “safe space”, working long hours, and having salads in the Berlaymont canteen for nutrition.

The best lawyers and customs experts from the commission and Whitehall were brought in on demand in order to draft legal text, the substance of which had been hammered out among a small coterie of civil servants led by Weyand on the EU side and Olly Robbins, Downing Street’s Brexit point-man, a mediaphobe who revelled in the tightness of the teams.

Weyand offered specific ambassadors among key member states, notably France and Germany, irregular briefings as the secretive talks developed.

When pressed for more information Weyand had a ready response.

“I can’t say much, it is all very sensitive,” she told concerned diplomats. “It is like a jigsaw. I can’t brief on a specific piece because it doesn’t make sense. It is only if you start putting them together that they start making sense. What I want to do is preserve those pieces so I can use them in the future. If I start to brief them now they will deteriorate. This is how we can explore what is possible”.

But the first phase of the tunnel talks ended in dramatic failure.

Robbins had returned to London on the Eurostar with his team, including Sarah Healey, the director general at the Department for Exiting the European Union, at about 6pm on the evening of Friday 12 October with some hope that there might be a workable deal there.

By the Sunday, with the Democratic Unionist party and members of the cabinet making obvious their unhappiness at what had been achieved, the Brexit secretary, Dominic Raab, was sent to Brussels to break the news to Barnier that there was not enough in the package to sell to parliament.

Theresa May spoke to the Commons that Monday to explain the proposal so far would have introduced a new commitment to negotiate, during the transition period after Brexit day, a temporary UK-wide customs union with the EU in order to avoid the “dislocation” of the country by creating a customs border in the Irish sea between Northern Ireland and Great Britain.

She confirmed the offer from the EU, first revealed by the Guardian, for an extension clause on the transition period to allow such negotiations to come to fruition if the 21 months was too short.

The EU had also insisted, however, on keeping in the withdrawal agreement their original “backstop” solution in which Northern Ireland alone would in effect stay in the customs union and single market just in case the negotiations on that mooted all-UK deal fail, she said.

May told MPs: “As I have said many times, I could never accept that, no matter how unlikely such a scenario may be.”

The question now was whether Weyand and Robbins could find some new jigsaw pieces in the tunnel.

“There is an evolution in the commission thinking,”, said one EU official ahead of the latest tunnel talks. “From what they originally proposed there is a shift towards the UK in agreeing to a UK customs-wide thing so, yes, flexible. But whether that flexibility will lead to agreement is something else.”

What has emerged in recent days is more than evolution, but a major leap by the commission that has many member states concerned and may yet be a source of trouble in the weeks to come.

By essentially including a fully operational UK-wide customs union in the withdrawal agreement, major issues that would have been negotiated in the transition period have been thrust into the last hectic few weeks left in the talks, EU sources said.

A customs union provides quota and tariff-free trade on goods. The EU has had to demand a huge array of “level playing field” commitments on state aid, to environmental, labour and social regulations to ensure that UK businesses cannot undercut European ones.

The EU had also wanted to use its leverage in the trade talks during the transition period to ensure continuity of fishing rights for European fleets in British waters after Brexit. But by giving away a customs union, including free trade in fish, that leverage would be lost.

Last Friday, in comments that sources say have been inaccurately seized upon by some as evidence of the EU rubbing its hands at the prospect of ensnaring the UK into a permanent customs arrangement, Weyand tried to persuade ambassadors that the British had not won out in the tunnel.

“Take a step back. What are we trying to achieve?” Weyand had asked the ambassadors, before answering: “What we are trying to achieve is to get the withdrawal agreement for citizens, financial settlement and transition and we are trying to get in the best possible position for the future negotiation.”

Agreement now on the difficult issues on state aid and environment regulations, among others, should be regarded as a “blessing in disguise”, Weyand said. A base-line had been set for the talks over the future relationship when it came to that after 29 March 2019. But it was a hard sell, sources said. Not triumphant. All eyes have been on the cabinet on Wednesday.

On Monday, EU diplomats suggest, the member states, particularly France, might send the commission back to demand harder commitments from the UK. There may be more drama yet to come.