Theresa May’s Brexit talks with Labour have been criticised as a “blind alley” as she came under intense pressure from 14 senior party figures to abandon the idea of a cross-party pact.

The former defence secretary Michael Fallon said the talks should be stopped, after he joined 12 other former cabinet ministers and Sir Graham Brady, the chair of the 1922 Committee, in warning No 10 against any deal that involved a customs union.

Fallon, who was forced to resign by May in 2017 for inappropriate behaviour towards women, said it would be better to stay in the EU than sign up to a customs union – a key demand of Labour.

“This is a blind alley taking us into a customs union,” he told BBC Radio 4’s Today programme. “We said we would leave a customs union very clearly at the time of the election. If you go into a customs union you can’t start pursuing independent trade deals.

“Far better to start concentrating on the other side which is the arrangements in Northern Ireland. The talks are clearly not getting anywhere. If they are going to include permanent membership of the customs union, then frankly we would be better off staying in the EU.”

May’s cabinet and Jeremy Corbyn’s shadow cabinet are both due to take stock of the talks on Tuesday, with neither side optimistic about the prospects for an agreement that could secure a majority in parliament.

Meanwhile, the prime minister’s Brexit negotiator, Olly Robbins, was travelling to Brussels for talks with EU officials on whether the political declaration could be changed, in an effort to break the deadlock.

The BBC said Robbins would explore how quickly changes could be made to the declaration if the government and Labour could come to an agreement.

In a letter to the prime minister, former cabinet ministers including Gavin Williamson, who was sacked this month over the Huawei leak, the former Brexit secretaries David Davis and Dominic Raab, and the former foreign secretary Boris Johnson, cautioned her against any deal involving Corbyn’s central demand of a customs union.

“We believe that a customs-union-based deal with Labour will very likely lose the support of Conservative MPs like us who backed the withdrawal agreement in March (in many cases very reluctantly), and you would be unlikely to gain as many Labour MPs to compensate,” said the letter, obtained by the Times.

“More fundamentally, you would have lost the loyal middle of the Conservative party, split our party and with likely nothing positive to show for it. No leader can bind his or her successor, so the deal would likely be at best temporary, at worst illusory.”

The fragility of May’s leadership, and the threat it presents to any deal being deliverable, has been discussed openly in the cross-party Brexit talks, which resumed on Monday.

Labour is demanding what it calls “entrenchment”, a legal underpinning for any promises made by the prime minister, to prevent a future leader unpicking them. In signing the letter, Johnson and Raab underlined Labour’s fears by making clear they would aim to sweep away any deal that involved a customs union.

The presence of Brady’s signature on the letter is also significant, as he is charged with the job of conveying backbenchers’ views to the prime minister. The 1922 Committee’s executive is expected to return for a third time this week to the issue of whether Tory leadership rules should be changed to allow a fresh challenge to the prime minister.

May had been keen to table the withdrawal agreement bill (WAB) before the European elections on 23 May, but the mood of Conservative MPs has been hardening against it, even without any customs union concession.

A number of Eurosceptic Conservative MPs were already planning to switch back to rejecting May’s deal, adding to the 28 leavers and six remainers from the party who held out against it last time. Tory MPs said support for May’s deal was “peeling off” as the prime minister’s power was ebbing away and they wanted to show their members and constituents they were serious about a hard Brexit.

The Eurosceptic Steve Baker indicated over the weekend that he would stand in the leadership contest if there was no other candidate who clearly opposed the withdrawal agreement, suggesting that Raab and Johnson could be outflanked on the right unless they take a position against it. If Baker does not stand, another of the 28 “Brexit ultras” is likely to put themselves forward.

Other Tory MPs suggested that May’s agreement was growing more unpopular daily as polls demonstrated the extent of support for Nigel Farage’s Brexit party, which is endorsing a no-deal position.

Such is the concern about support draining away from the deal that ministers have discussed breaking up the legislation into what one government source called “bite-sized WABs”, in an effort to ratify less controversial aspects.

The aim would be to demonstrate some progress in delivering Brexit, ideally before polling day, though ministers are increasingly pessimistic about meeting that deadline. Such a move would probably be regarded with exasperation by Britain’s EU negotiating partners, who signed off on the withdrawal agreement as a package.

Cabinet is expected to discuss whether to press ahead with any more Brexit talks with Labour or move to a series of parliamentary votes with a binding outcome – or “definitive votes”.

The cross-party negotiations broke up on Monday night with both sides once again signalling no substantive progress had been made.

Like May, Corbyn is constrained by his backbenchers. He came under pressure from MPs at the weekly meeting of the parliamentary Labour party on Monday night over what several claimed was the party’s fuzzy stance on Brexit and lacklustre approach to the European election campaign.

The Ilford North MP, Wes Streeting, said: “We’re trying to please everyone and instead are pleasing no one.”