As Labour’s Brexit negotiating team headed for the Cabinet Office buffet after a morning of intense talks with their Tory counterparts on Thursday, one concern was already nagging away at them. The discussions had been good-natured, thorough, serious – but had also felt a bit like a re-education programme.

After grabbing their sandwiches and cake, Keir Starmer, the shadow Brexit secretary, and other members of Labour’s team slipped off to a small room to meet in private. They had all come to the same conclusion. As well-intentioned as the talks had been, they were essentially being told, in the most polite terms possible, that they should simply reconsider backing May’s clever deal, which they just hadn’t quite understood.

The offer of cross-party talks by the prime minister on Tuesday night shocked MPs in both parties. The usually lively WhatsApp group of Tory MPs went quiet for four hours, before someone tentatively asked: “Is anyone there… ?” Meanwhile, the Labour figures primed to take part in the talks hoped this was the moment May had accepted reality – that, with no majority for her deal, she had to entertain some of Labour’s demands and alter the “political declaration”, which sets out the broad aims of Britain’s future relationship with the EU.

There was no obvious breakthrough in initial talks between Jeremy Corbyn and May on Wednesday, which took place over less than two hours in the prime minister’s House of Commons office. As is her style, May gave little away as the Labour team set out their problems with the current deal and raised the issue of a confirmatory vote.

Other MPs to hold a meeting with May last week found the same issue. When Labour MP Rupa Huq met the prime minister, she attempted to break the ice by talking about the TV programmes Bodyguard and Fleabag. May had seen neither. The prime minister and her aide chanted in unison that she “prefers [US crime drama] NCIS”.

Putting the lack of an obvious shift down to the prime minister’s passive style, Corbyn’s team pinned their hopes on Thursday’s longer and more detailed talks, taking place in the cabinet office and chaired by David Lidington, May’s pragmatic de facto deputy.

Starmer, Labour’s chief whip Nick Brown and the shadow business secretary Rebecca Long-Bailey led the team. The morning was largely dominated by the issues that risk causing friction at the border and damaging trade – leaving the EU’s customs union and single market. Olly Robbins, May’s chief Brexit negotiator, gave a detailed account of how May’s current deal was, in effect, a customs union – and would also see Britain align with the single market.

The government’s team was effectively stating that their deal contained measures that they had publicly said they were against. Labour’s team asked how Britain’s future EU deal could be altered and shaped. The queries were met with detailed and calm explanation about the EU’s position and what could and had been attempted. By lunch, Labour had detected a reluctance to actually make any significant changes.

Gavin Barwell, No 10 chief of staff, leaves the Cabinet Office in Whitehall. Photograph: Mark Thomas/Rex

“They were essentially setting out why the deal was very good and how we hadn’t quite understood how good it was,” said one source familiar with the talks. “How, if you look carefully, it is a customs union and it is alignment with the single market. It was a really good, interesting technical exposition of the deal – but it was pretty clear they were selling their deal rather than explaining how they would change it.”

After lunch, national security and workers’ rights were examined –with Labour asking for worker protections to be beefed up. Discussion kept coming back to one point, made repeatedly by Starmer – how on earth could Labour have any confidence that a new Tory leader, perhaps Boris Johnson or former Brexit secretary Dominic Raab, would honour anything agreed at these talks? It is also the latest argument being deployed by campaigners for a second referendum to dissuade Labour MPs from backing any deal.

Then the Downing Street team made what they thought was a big offer. They said that to make Labour feel like it had some control over the future trade deal secured with the EU, they would accept a plan drawn up by Labour backbenchers Lisa Nandy and Gareth Snell, handing parliament some control over the parameters of future EU talks. They even offered to enshrine the idea in law.

The Tory team were baffled that Labour did not see this as a big win. For Labour’s team, the offer amounted to very little – it still required Labour to vote through a deal with no guarantees about the future, while handing parliament a say did not guarantee that Labour’s demands would be heeded.

The meeting ended on another difficult note as Labour brought up the issue of a second referendum – or a “confirmatory vote” on May’s deal. According to some in the room, it was the most uncomfortable moment, with Stephen Barclay, the Brexit secretary, making it clear how difficult the idea would be for the government.

When a government document on the talks arrived with Labour on Friday – making no indication that the government was willing to alter the political declaration in May’s deal – Labour decided to speak out about the apparent unwillingness to compromise. That prompted a denial from Downing Street, with Philip Hammond, the chancellor, stating that there were no “red lines” in the talks.

Both sides insisted on Saturday night that technical discussions were taking place and talks would continue. Figures on both sides also said that there was a more difficult issue that had not yet had serious discussion: if no cross-party deal can be reached, can the two parties agree to a series of votes in parliament that will solve the Brexit impasse, once and for all?

Senior figures on both sides suggested that the idea, raised by the prime minister as part of her invitation to Labour on Tuesday, had not yet been raised. “There’s a sense that the government is backing away from that,” said one person involved. “Once you hold those, you have no idea where it goes.”

The idea is fraught with danger for second referendum campaigners, too. “We don’t want a situation where a second vote is accidentally killed off,” said one insider. “The best thing would be a long extension [to Britain’s EU membership].”

Theresa May’s director of communications, Robbie Gibb. Photograph: Steve Back/Getty Images

Tory team

Robbie Gibb Downing Street director of communications and former BBC editor. Accused by MP Nick Boles of being in favour of hard Brexit.

David Lidington Theresa May’s de facto deputy and former Europe minister who is seen as a non-divisive figure by Labour’s team, but distrusted by Tory Brexiters.

Steve Barclay Brexit secretary who was a surprise choice after Dominic Raab quit. Wants to keep open the option of leaving the EU with no deal.

Gavin Barwell Former Croydon MP and the prime minister’s amiable chief of staff. Has been giving presentations on May’s deal in the hope of gaining supporters.

Labour team

Seumas Milne Strategy and communications director, who has significant influence on the Labour leader. Regarded as a Eurosceptic.

Nick Brown Party veteran and Labour chief whip, who has the job of working out what most Labour MPs can accept. Respected by the Tory team.

Rebecca Long-Bailey Shadow business secretary and protege of John McDonnell, who leads on Labour’s concerns about workers’ rights after Brexit.

Keir Starmer Shadow Brexit secretary who has a key role in Labour’s cross-party talks and has argued in favour of keeping a second referendum on the table.

• This article was amended on Sunday 7 April to correctly identify the person on the right of the main picture as Anjula Singh, and not Rupa Huq as we had it previously.