The former Tory leader Iain Duncan Smith has called on Theresa May to resign as prime minister next month, saying the timetable for her departure should still stand despite her failure to pass a withdrawal agreement.

“I know that the prime minister has already said she’s going. She said she would go as and when the agreement was ratified, which was looking at around about May, June. I think those dates still stand,” he told Sky’s Sophy Ridge on Sunday.

“I think that what the PM has to do is aim everything now towards departure before the Euros [elections] which would then allow her to step away having done what she said she would do, getting the UK out of the European Union one way or the other and then we can have another leadership election and pick a new leader, which is the way it has to be.”

His intervention came as two former chairmen of the 1922 Committee of backbench Tory MPs said party rules could be changed to allow a leadership challenge sooner than December.

Writing in the Sunday Telegraph, Lord Spicer and Lord Hamilton of Epsom said the 12-month rule on no-confidence votes could be changed if MPs agreed to do so. “Conservative MPs are responsible for their party. If they wish to change these rules there is nothing standing in their way,” the pair wrote.

Graham Brady, the chair of the committee of backbenchers, said he believed such a change would be possible but said he was “less certain that it would be possible to change the rules during the current period of grace which was initiated with the triggering of a confidence vote on December 12 last year”.

May faced a vote of no confidence in December after 15% of Conservative MPs wrote to Brady to request the vote, and won by 200 votes to 117.

The party’s former chairman Sir Patrick McLoughlin warned those agitating for a swift leadership change that installing a hard Brexiter in No 10 would not necessarily solve the party’s electoral problems.

“Defining ourselves as the Brexit party, pursuing the hardest form of Brexit with a parliament that will not deliver it, is a recipe for paralysis in government and suicide with the electorate,” he wrote for the Sunday Times. “We are the Conservative party, not the Vote Leave party.”

Recent polling has predicted dire results for Conservatives. The latest Opinium poll suggests the party has dropped six points over the past fortnight, giving Labour a seven-point lead. The support is the lowest the Tories have scored since the European elections in May 2014, almost five years ago, according to the pollster.

The poll shows the party losing ground with both leave and remain voters. Support from pro-EU Tory voters has fallen from 23% to 17% and the party has also fallen by 7% with leave voters. More than a fifth of leave voters now say they would vote Ukip.

Duncan Smith said many in the party were deeply concerned about the polling. “It was on the 29th when we didn’t leave; that’s when this has all gone wrong. Up until then, people were prepared to give Theresa May the benefit of the doubt,” he said.

“The big problem was as soon as we didn’t leave, you could see all the poll ratings start to crash. And it’s wholly linked to the fact that to leave or remain, they were all expecting us to go, and when we didn’t go it looked like a complete breach with the pledge that we had made, and that’s a disaster for a political party.”