A Conservative MP has tabled a motion of no confidence in John Bercow after the House of Commons Speaker said Donald Trump should not be allowed to address parliament because of his racism and sexism.

James Duddridge, a former Foreign Office minister, told Sky News that there was a great deal of support on the Conservative benches for removing him.

“He has overstepped the mark a number of times but this most recent incident – where he used the Speaker’s chair to pronounce his views on an international situation in some quite detailed and lengthy manner – is wholly inappropriate and it means that he can no longer reasonably chair, as Speaker, any debate on those subjects,” he said. “This has been happening more and more often from this modernising Speaker. This is perhaps the straw that has broken the camel’s back.

“He doesn’t really understand the degree of the anger in the House of Commons, the distrust in his role as Speaker of the House of Commons, and I expect over the recess – because parliament now shuts down for one week – over that week the number of MPs speaking out either publicly or privately to journalists will increase and increase and it will be known his position is untenable, perhaps even to the point that he doesn’t return on the Monday.”

The government has said the future of the Speaker is a matter for parliament after Duddridge wrote to the prime minister asking her to confirm that she would offer a free vote if a vote of no confidence was called.

Bercow, a Conservative whose role is politically neutral, has long been disliked by some of his own party in parliament but there is unlikely to be enough support in the Commons to remove him.

He survived an attempt to oust him orchestrated by the Conservative frontbench under David Cameron and this time would be very likely to have the backing of opposition parties to continue until he steps down voluntarily, most likely towards the end of this parliament.

Theresa May confirmed during her recent trip to the White House that Trump had accepted an invitation from the Queen to pay a state visit to the UK later this year. But the plan has caused disquiet among some MPs and sparked protests.

The Speaker intervened on Monday when he made clear that he was always against the idea of Trump making a speech in the same hall that Barack Obama did in 2012, but that recent policies had left him even more determined to block the move.

“After the imposition of the migrant ban by President Trump I am even more strongly opposed to an address by President Trump in Westminster Hall,” Bercow told MPs. “I would not wish to issue an invitation to President Trump.

“I feel very strongly that our opposition to racism and to sexism and our support for equality before the law and an independent judiciary are hugely important considerations in the House of Commons.”

Bercow has already apologised to his counterpart in the House of Lords, Norman Fowler, for not consulting him before expressing the view that Trump should be allowed to address parliament during the upcoming state visit.

He defended himself to the Commons on Tuesday, saying he was “commenting on a matter that does fall within the remit of the chair”.

Earlier on Thursday, the Speaker received support from Labour in the Commons, with the opposition urging ministers to reject calls for a vote of no confidence.

Valerie Vaz, the shadow leader of the Commons, urged her opposite number to “confirm that the government will not support any attempts to act on the letter to the prime minister about comments made on a point of order in this chamber”.

David Lidington, the Commons leader, did not respond to the question but said the government had to deal with the US president as he was democratically elected, despite strong feelings on the matter.

He said: “Whatever view any of us as individuals might have on any particular leader of another country, the reality is that governments have to deal with other governments in the world as they exist and particularly with elected governments who are able to claim a mandate from their own people.”