John Bercow is facing another attempt to oust him as Speaker after a leading Conservative MP began collecting signatures for a no-confidence motion.

Crispin Blunt, a former chair of the House of Commons foreign affairs committee, wrote to fellow MPs saying he wanted to table an early day motion saying Bercow was no longer impartial.

In the letter, he claimed to have some frontbench Conservative support for the move against the Speaker, who has enraged the government in recent months by refusing to let MPs vote repeatedly on Theresa May’s Brexit deal after their rejections of it.

However, Blunt said he would only table the motion if he got 100 signatures from MPs, in order to give cover to those who backed it against “retribution” from the Speaker. Labour MPs said Blunt had approached them as well, but he is thought to have few, if any, opposition signatures.

The Conservative MP made the move following reports that Bercow wanted to delay his departure as Speaker while Brexit remains unresolved. Pro-leave MPs have accused him of being biased against Brexit, but some hardliners also applauded him for refusing to let May’s withdrawal bill back into the Commons without substantial changes. Despite the reports that he may want to stay on, it is understood Bercow has told some friends he is still thinking of naming a date for his departure this summer.

Crispin Blunt. Photograph: Anthony Devlin/PA

In a letter to colleagues first reported by the Guido Fawkes blog, Blunt said: “This is the most important house matter. We all know the truth and precedent is currently an unreliable guide! We are in totally unprecedented territory. If colleagues are unable to say JB is partial, then he will feel able to continue. But we won’t have stood up for our most basic standards and can blame no one but ourselves.”

The motion would state: “This house … believes that it is impossible for this house to sustain belief in the impartiality of the right honourable member for Buckingham; and therefore has no confidence in the Speaker.”

Recent attempts to oust the Speaker have ended in failure, including one in 2017 that garnered only a handful of signatures. A previous effort in 2015 backed by William Hague, then leader of the Commons, and the then chief whip, Michael Gove, came closer to succeeding, after they brought forward a motion to ensure there was a secret ballot to elect the Speaker in the next parliament. This was defeated by 228 to 202.