Theresa May has come under more fire from MPs within her own party, with the former Brexit secretary Dominic Raab suggesting the prime minister has failed to stand up to a bullying European Union over the withdrawal deal.

Meanwhile the Conservative MP for Richmond, Zac Goldsmith, has revealed he has written a letter of no confidence in May to the party’s 1922 Committee.

Raab, who stepped down as Brexit secretary on Thursday saying he could not accept the terms of the deal done by the prime minister, told the Sunday Times the UK should demand an agreement that allows it to unilaterally leave any customs union.

He spoke as an opinion poll for the Observer showed Labour had opened up a three-point lead over the Conservatives, with leave supporters in particular deserting the Tories.

Raab said: “If we cannot close this deal on reasonable terms we need to be very honest with the country that we will not be bribed and blackmailed or bullied and we will walk away. I think there is one thing that is missing and that is political will and resolve. I am not sure that message has ever landed.”

Both May and Raab are due to take their arguments on to television screens on Sunday morning.

Play Video 1:16 Threshold for confidence vote not reached, says prime minister – video

Last week saw the departure of Raab and the work and pensions secretary, Esther McVey, plus the launch by hardline Tory leaver Jacob Rees-Mogg of a high-profile insurrection on the backbenches to remove May from office.

Goldsmith, the failed Tory London mayoral candidate in 2016, revealed he has joined those who have sent a letter of no confidence in May to Sir Graham Brady, chairman of the 1922 Committee of backbench MPs.

Writing in the Sunday Telegraph, Goldsmith said that under the PM’s plan “in effect Britain would remain in the EU, but without having any say”. He added: “Had that been the choice, I personally would have voted to remain.

“The withdrawal agreement we have been presented with is unacceptable to leave and remain voters alike. It has close to zero chance of making it through parliament and with only five months remaining, we cannot afford to waste any more time on it.”

In an interview with the Mail on Sunday, Amber Rudd, who was brought back into the cabinet to replace McVey, condemned attempts to force a no-confidence vote, warning it could make the government look “unhinged”. “I hope they go back into their corners and we can get on with doing what we’re expected to do, which is delivering a Brexit I hope will protect the economy,” she said.

“They might have gone off a little early because it feels to me they’re rowing back.

“What could be madder at this stage, with seven days to go [to the Brussels summit] to undermine the prime minister? Such a mistake.”

But in a sign the Tory civil war over Brexit was not abating, the Middle East minister, Alistair Burt, warned rebels the “consensus” that pro-EU MPs should reluctantly respect the 2016 referendum result could break down if May were toppled.

Alistair Burt (@AlistairBurtUK) Be very clear. If an agreed deal on leaving between the Govt and the EU is voted down by purist Brexiteers, do not be surprised if consensus on accepting the result of the Referendum by Remain voting MPs breaks down. Parliament will not support no deal. https://t.co/dx5CBigNDs

Burt, whose role spans the Foreign Office and Department for International Development, attacked a tweet by Steve Baker, the deputy chairman of the European Research Group, in which the latter promoted a lawyer’s legal case against the agreement reached with Brussels.

Brussels tells Theresa May - delaying Brexit will cost UK £10bn Read more

Burt wrote: “Be very clear. If an agreed deal on leaving between the government and the EU is voted down by purist Brexiteers, do not be surprised if consensus on accepting the result of the referendum by remain voting MPs breaks down.

“Parliament will not support ‘no deal’.”