Ministers are set to get a free vote on any confidence motion in Speaker John Bercow after he infuriated Tory MPs by declaring he voted for Remain.

Tory rebels are determined to dislodge Mr Bercow from his position and want a vote to be called amid claims the 'scoresheet was mounting up' against him.

Anger against the Speaker spilled over after it emerged he told a group of students at Reading University that he backed Remain and supported immigration.

Cabinet minister David Lidington stopped short of full support for the Speaker in a round of TV interviews - but did insist it was a matter for MPs and not ministers. Tory sources confirmed a free vote would be allowed.

Were a vote to be called, the support of ministers would increase the political danger to Mr Bercow even if he had enough support to win it.

The uprising against Mr Bercow - which currently amounts to just a handful of Tories but which could become politically toxic - began after his tirade against US President Donald Trump speaking in Parliament during a state visit this year.

John Bercow, pictured, may have broken Parliamentary rules over impartiality as Speaker of the House after he was filmed admitting he voted for the Remain campaign

Commons leader David Lidington, pictured on the BBC Andrew Marr programme today, distanced the Government from the row over Mr Bercow, warning he must have support across Parliament to do his job

According to The Telegraph, Mr Bercow was filmed revealing how he voted in the EU Referendum at an 80-minute question and answer session on February 3, three days before he made his comments about President Trump.

The paper said Mr Bercow claimed 'promises were made that could not be kept' during the campaign and added EU migration was a 'good thing' and immigration from other parts of the world was a 'positive far more than a negative'.

It has led to MPs stating he 'cannot come back to the chair' because he is 'in clear breach of guidelines over independence'.

WILL THE MOTION SUCCEED IN KICKING BERCOW OUT? Tory MP James Duddridge's Early Day Motion has only one signature - his - and is highly unlikely to ever be debated. He is hoping it will become politically damaging through weight of support. In 2009, when Douglas Carswell tabled a similar motion of no confidence in the then Speaker Michael Martin, 22 MPs signed prompting him to quit. Mr Duddridge hopes that by increasing the pressure on Mr Bercow, the Speaker will also decide it is time to step down from his powerful role. Advertisement

Mr Duddridge, who tabled the motion of no confidence in the Speaker over his Trump comments, told John Pienaar the speaker is 'no longer impartial'.

Speaking on BBC Radio 5live's Pienaar's Politics, he said: 'I think there will be a vote of no confidence and I think he will go.

'There's absolutely no way Speaker Bercow can sit in the chair on European issues.

'When you become Speaker you must be impartial. He's no longer impartial, he's no longer able to continue to do the role, which is why I think the House will vote him down in a vote of no confidence. In reality he may see the lie of the land and go before he's pushed.'

He added he had been 'amazed' by the number of people to privately voice to him their support for his motion.

Because the Commons is on recess, nobody can sign Mr Duddridge's motion - which currently only has his own name attached - until February 20.

Mr Duddridge's motion alone will not be enough to trigger a Commons vote on the Speaker but it could produce the momentum necessary to make one inevitable.

Tory MP James Duddridge, pictured on Sky News, has tabled a motion of no confidence in the Speaker, branding Mr Bercow's attempt to interfere in President Trump's state visit 'wholly inappropriate'

Former culture secretary John Whittingdale said the number of supporters was 'mounting up'.

He told ITV's Peston on Sunday: 'John was elected with a very firm pledge that he wouldn't stay for more than eight or nine years, and we are pretty much getting close to the end of the that period.

'So, I wouldn't expect him to stay for much longer.'

Mr Lidington, the leader of the Commons, distanced Theresa May's administration from the row in an interview on the BBC One Andrew Marr programme.

He said: 'There will be strong reaction among some MPs to what he said at Reading, particularly after what he said about the state visit earlier in the week. Ultimately, the Speaker has to command the confidence of the House of Commons as a whole.

'John has his very strong supporters as well as his strong critics in the House of Commons, but we shall have to see how members as a whole respond.

'It is really important for the very independence of the Speaker's office that the Speaker, whether they start as a Conservative MP, a Labour MP, or whatever, is independent of Government.

'Speakers, if anything, should be towards the people who are not in Government, as, actually John Bercow probably has done in the way that he has used urgent questions that we have found inconvenient.'

Labour deputy leader Tom Watson offered a strong signal of support, describing Mr Bercow as 'one of the great Speakers the House of Commons has seen' on the BBC.

The Speaker, pictured, is already facing a vote of no confidence over his attack on Donald Trump

But a spokesman for the Speaker told the Telegraph he had been 'even-handed' in his official approach to EU matters.

The spokesman said: 'The Speaker's impartiality is required on matters of debate before the House, and he has been scrupulous in ensuring that both sides of the argument are always heard.'

He added: 'How he voted and whether or not he said how he voted after the event is completely irrelevant. It had absolutely no bearing on his impartiality.'

The latest row comes after it emerged Mr Bercow has told friends he wants to stay on as Speaker until 2020, despite the formal moves to force him out after his attack on Mr Trump this week.

Mr Bercow pledged that he would serve for nine years when he took over as Speaker in June 2009. Under this timetable he would step down in June next year.

But he has refused to commit publicly to a departure date in recent years. And a senior Tory source said he was now canvassing opinion about the idea of staying in post until the next election in May 2020 – almost two years beyond his promised exit date.

'John is being very vague about it,' the source said. 'He did promise it would be nine years but it now appears to be 'about nine years'. He still has a good relationship with his local Conservative Association and is suggesting he wouldn't want to put them through a by-election campaign.

'In the end, it's up to him. He promised he would go after nine years but there is very little anyone can do about it if he decides to changes his mind.'

Labour deputy leader Tom Watson told the BBC's Andrew Marr that Mr Bercow was 'one of the great speakers the House of Commons has seen.

Ministers were privately infuriated by the 'insult' to Mr Trump, who was invited by Theresa May to make a state visit to the UK in June.

On Thursday, former Tory minister James Duddridge took the rare step of tabling a motion of no confidence in Mr Bercow, saying he had 'overstepped the mark'.

Mr Duddridge said the attack on the US President was 'the straw that broke the camel's back', following long-running concern about anti-Tory bias.

'It was wholly inappropriate and it means that he can no longer reasonably chair, as Speaker, any debate on those subjects.'

Mr Bercow, pictured with wife Sally, has also denied stopping the Commons being warned about a police probe into Labour MP Keith Vaz

He added: 'I have got absolute confidence that a majority of MPs will be in the 'he's not doing a good job and should go' category.

'How many of those will go as far as voting in a vote of no confidence and how many will sign up to an EDM, I genuinely don't know.'

Fellow Tory Alec Shelbrooke said that although he disagreed with Mr Trump's 'discriminatory' travel ban, Mr Bercow had gone too far.

He said he would back the motion to eject him, saying Mr Bercow had 'politicised the office of Speaker and his position is untenable'.

Tory Andrew Bridgen became the third MP to call publicly for Mr Bercow to quit yesterday, saying: 'Speaker Bercow's position is untenable, he needs to be replaced before President Trump's state visit.'

According to The Times, donations to help the Speaker keep his parliamentary seat reportedly included a sum from Narinder Chadha - chair of the management board of Mr Vaz's Silver Star charity.

However a spokesperson for Mr Vaz told MailOnline the newspaper's story was 'wrong' and that Mr Chadha denies having spoken 'this year' to The Times.

The newspaper reported an alleged conversation with Mr Chada in which he is said to have confirmed that he gave the money after a request from the MP for Leicester East.

Mr Vaz's representatives deny this saying that Mr Chada will be complaining to IPSO over the Times article.

In total, the Speaker accepted £41,000 from donors thought to be linked to Mr Vaz, according to the newspaper.

The Times claimed Mr Bercow refused to tell Mr Vaz to resign as chair of the Home Affairs select committee after Tory MP Andrew Bridgen privately warned him the Labour MP was being investigated by police - an investigation Mr Vaz later said he knew nothing of in September 2015.

A spokesman for Mr Bercow, pictured, said how he voted in the Referendum had 'no bearing on his impartiality'

The Speaker later interrupted Mr Bridgen when he attempted to talk about the allegation in the Commons, according to The Times.

A spokesperson for the Speaker told the MailOnline that 'a personally defamatory attack on another Member[ was not allowed] on the floor. The Speaker allowed....[Mr Bridger to speak] on matters of public record, and stopped him when he sought to embark on other allegations, and again when he sought to reveal the contents of confidential correspondence with the Speaker.'

Mr Vaz did eventually step down as chairman of the Home Affairs Select Committee last year after he was caught meeting two male prostitutes.

This scandal was said to be unrelated to the police investigation in 2015.

A spokesman for Mr Bercow added the role of Speaker did not give him the right to tell an elected member of a select committee to step down.