Parliament is facing a day of further Brexit deadlock after Theresa May swung the government’s weight behind an amendment that would send her back to Brussels to demand an alternative to the Irish border backstop, splintering Conservative support.

The chances of the amendment, championed by the senior backbencher Sir Graham Brady, are on a knife-edge after Tory Brexiters split over whether they should back the change, while pro-remain MPs suggested they would vote against.

Jacob Rees-Mogg, the chair of the hard-Brexit European Research Group (ERG), emerged from an earlier meeting with MPs saying the group would not back the move. An hour later, other pro-Brexit Tories left a meeting with the prime minister sounding less certain of their opposition.

Brady’s amendment would approve May’s withdrawal agreement, but says the government must replace the backstop with “alternative arrangements” – a change that senior EU figures have repeatedly suggested would not be acceptable.

May addressed Tory MPs in parliament on Monday night, spelling out her support for Brady and urging them to support the amendment.

According to sources, May told the meeting that if the government was going to have “a reasonable chance” of convincing Brussels of the seriousness of the opposition in parliament, then it was a vote that mattered, rather than words.

Boris Johnson, who has been at the forefront of demands to reopen the text of the withdrawal agreement to remove the backstop entirely, was publicly rebuked by the prime minister after he asked how the amendment would lead to meaningful change.

“We won’t know until you support us, Boris,” the prime minister retorted, prompting some cheers in the room. “I am happy to battle away. Get behind me and we will do it together.”

A friend of Johnson said there had been attempts after the meeting by the No 10 chief of staff, Gavin Barwell, to reassure Johnson that May did intend to reopen the text of the withdrawal agreement.

It is likely, however, that pro-Brexit MPs currently opposing the Brady amendment will need public reassurance from the prime minister that this is what she is seeking. Senior Brexiter sources said similar private reassurances had also been given to ERG MPs by the chief whip, Julian Smith.

Yet many were spooked by Brady’s comments to the BBC on Sunday night that the change would not require rewriting the agreement, but that a “codicil” with the new arrangements would suffice.

Their concern was compounded after the prime minister declined to repeat those assurances in the meeting, one Brexiter source said. “That is the main reason why the bulk of the ERG are minded not to support this,” the source said. “If they vote for this, then they give up leverage for that demand to reopen the text.”

May was offered a lifeline late on Monday night as senior Tory Brexiters and remainers including Rees-Mogg and Nicky Morgan threw their weight behind a surprise compromise, backed by ministers, which would involve a redrafting of the backstop and an extended transition period until 2021.

The initiative has been called the “Malthouse compromise” after the housing minister Kit Malthouse who coaxed the two warring factions into attempting talks. However, it is still unclear whether the prime minister is prepared to pursue the offer of compromise or if the plan is workable.

In a WhatsApp message to Tory MPs, Morgan said the plan “provides for exit from the EU on time with a new backstop, which would be acceptable indefinitely, but which incentivises us all to reach a new future relationship. It ensures there is no need for a hard border with Ireland”.

The Conservative party chair, Brandon Lewis, confirmed that the Tories would be whipped to support Brady’s amendment to a government motion on Tuesday night if it is selected by the Speaker.

The success of another amendment tabled by Yvette Cooper, which would pave the way for legislation to delay the article 50 process, also looked uncertain on Monday night as senior Labour figures including Jon Trickett voiced doubt about the message it would send to voters.

“I feel that it may look to people as if we’re trying to somehow remove the earlier decision, which was to Brexit,” he said.

Without the explicit support of the Labour leadership, the amendment may struggle to win a majority. However, several Labour MPs said they had been reassured that the leadership would swing behind Cooper’s move on Tuesday.

The doubts about the backing for both amendments could potentially lead to parliament voting down all possible routes proposed by MPs to break the Brexit impasse.

MPs will vote on a number of amendments, yet to be selected by the Speaker, John Bercow, to a government motion that May was obliged to table after the heavy defeat of her Brexit deal. Government sources said they remained highly uncertain whether Bercow would select Brady’s amendment for a vote.

Before the vote, the shadow Brexit secretary, Keir Starmer, said the prime minister “has offered up no plan B or any credible solution to break the impasse” and demanded that the government spell out the next steps.

“We now need to see action from the government, not further delay or warm words,” he said. “The prime minister needs to spell out precisely what she plans to negotiate with the EU, a timeline for delivery, and when parliament will next have a say on the terms of any deal the government has negotiated.”

May is likely to make a new statement to parliament by 13 February on the next steps for the Brexit process.

At their cabinet meeting before Tuesday’s vote, cabinet ministers will urge the prime minister to set out at the dispatch box when MPs will have the chance to vote again on her withdrawal agreement. “She needs to make a strong and clear statement,” one cabinet source said.

The ERG will meet at 6pm on Tuesday night, an hour before the vote, to consider their final verdict on Brady’s amendment and any additional government reassurances.

Loyalist Tories piled in behind the amendment after the prime minister gave it her backing at the meeting, queueing up to add at least 20 names to it.

Yet some other party moderates remain deeply sceptical of the purpose of the amendment and what May can use it to achieve in Brussels. “If Brexiteers buy this they’re a total pushover,” one Tory MP said. “It may as well have said: ‘I’d like world peace.’ What it does is underline that we don’t have an alternative.”

May will also face a core group of rebels from the pro-remain group in her party, who may sink the amendment themselves even if the Brexiters eventually back it.

Heidi Allen, one of the Tory MPs backing a second referendum, said: “I cannot believe our prime minister is prepared to renege on an internationally agreed treaty, via a backbench-sponsored amendment. I won’t be voting for it and many of my colleagues won’t either.”

Those behind Brady’s amendment had also hoped to attract the support of the DUP. However, on Monday the party’s Brexit spokesman, Sammy Wilson, urged a hardline approach. “The real chaos when it come to the backstop is in Brussels and Dublin,” he said. “Now is the time for the prime minister to exploit the cracks which are emerging in the illogical position of the EU and the Irish.”