Dominic Raab has conceded that Conservative backbenchers have become jittery about the status of the Brexit negotiations but insisted “now is the time to play for the team” as speculation about Theresa May’s leadership continues.

The Brexit secretary said the Tories needed to hold their nerve after weekend reports that hostile backbench MPs would try to force a leadership challenge. Forty-eight MPs are needed to force a vote of confidence in the prime minister.

“We’re at the end stage of the negotiations. I think it’s understandable that there jitters on all sides of this debate. We need to hold our nerve; the end is in sight in terms of a good deal – the prize that we want: a good deal with the EU,” Raab said.

“Colleagues should wait and see what that looks like. It won’t be a question of a fait accompli. They’ll have their a full say on it. That’s what meaningful vote is all about. We won’t want to bring something back that we aren’t confident is a very good deal for the United Kingdom. And now is the time to play for the team.”

When asked if May was hanging by a thread, Raab joked that it was a pretty strong thread and said there were reports every week that nearly 48 Tory MPs had privately written to the chair of the backbench 1922 Committee demanding a vote of confidence.

Raab also appeared to fire his own warning shot at No 10 when he said he was confident of getting a good Brexit deal, but that “there does come a point where you can be compromised yourselves by the compromises you make”.

Brexit phrasebook: a guide to the talks' key terms Read more

The cabinet minister set out the negotiating red lines on BBC1’s Andrew Marr Show on Sunday. When it comes to any backstop agreement, he said, “Northern Ireland can’t be separated from the rest of the UK in customs or regulatory terms and we cannot have a situation where we are stuck indefinite limbo.”

Raab was speaking as speculation about May’s leadership continued. Hard Brexiters were particularly unhappy about suggestions that emerged from the EU summit last week that the UK could sign up to an extension of the post-Brexit transition period, in which Britain would remain in the customs union and single market for a few months into 2021.

The former Brexit secretary, David Davis, touted by some as a possible successor to May, wrote in the Mail on Sunday that “even the most charitable verdict on last week’s Brexit talks in Brussels can hardly describe them as a success”, because May appeared to endorse extending the transition period beyond December 2020.

Raab said he was open-minded about the possibility of a short extension, “let’s say three months” if that helped eliminate the need for a backstop in which Northern Ireland or all of the UK would remain in the customs union if negotiations on a future UK-EU free trade deal were not concluded by the end of 2020.

“It’s an obvious possible route, as long as it is short and we know how we get out of it. And it has to solve the backstop issue so that falls away as a possibility,” Raab said.

The Brexit secretary told Marr he was unable say when he would be returning to Brussels to meet with the EU’s chief negotiator, Michel Barnier, which would be interpreted as a sign that a final Brexit deal is close.

Officials from both sides will continue the talks this week in the hope of a breakthrough after last week’s summit, in which EU leaders agreed they would try to find a way through to striking a deal with May.

Writing in the Sunday Times, Johnny Mercer, a backbench MP, said he “cannot continue to support an administration that cannot function” on issues that affect people’s lives such as Brexit, the Windrush immigration crisis and the Grenfell Tower fire.

Mercer clarified his remarks on Sunday, saying he did “absolutely not” want May to quit and insisting that his piece, which did not mention the prime minister, was intended to argue for a positive direction for government policy. Mercer described the government last week as a “shitshow” that he would not vote for if he wasn’t an MP.

There were also complaints about some of the language used anonymously by May’s critics in the weekend papers. One told the Sunday Times the prime minister had entered “the killing zone” this week ahead of a crucial meeting of the Conservative backbench 1922 committee on Wednesday, where May is due to speak.

Another told the Mail on Sunday May should “bring her own noose” to the meeting. Nicola Sturgeon, the SNP leader, said: “Language like this debases politics. Get a grip, Tories.”

Keir Starmer, the shadow Brexit secretary, said there was a “real lack of confidence” that May could bring back “anything by way of a good deal” because the government was so divided. Also appearing on Marr, he said Tory infighting seemed “to have gone to another level”.

There are 315 Conservative MPs. Under party rules May needs to win the support of just over half – 158 – to win a vote of no confidence, but many believe that if more than 100 MPs were to vote against her she may not be able to continue as prime minister.