London (CNN) Speaker John Bercow is bowing out of the UK Parliament -- and he's leaving with a volley of parting shots for his many critics.

In a wide-ranging interview with CNN to mark the end of his term, Bercow, who has been accused of favoring the "remain" side of the Brexit debate , insisted that he had always acted as an impartial referee

And he had some stern words for lawmakers -- including members of Prime Minister Boris Johnson's Cabinet -- who have sought to portray Parliament as frustrating the will of the British people as expressed in the 2016 Brexit referendum in which the UK voted to leave the European Union.

In the interview with CNN's Bianca Nobilo, Bercow:

Bercow highlighted comments by Attorney General Geoffrey Cox, who called Parliament a "disgrace" in a recent debate for repeatedly failing to vote for a Brexit deal. Bercow said he would defend the right of ordinary lawmakers to hold the UK government to account "no matter how much cheap abuse, calumny and vituperation are directed to Parliament."

With just over three weeks to go until the latest Brexit deadline, Bercow speaks with CNN.

Johnson has been accused of whipping up division with "Trumpian" rhetoric and seeking to frame Parliament as frustrating the will of the British people to leave the EU.

The attacks had no effect on him, Bercow said. "I'm entirely unmoved by some of the more downmarket attacks on Parliament that have been launched in recent times. They are unworthy, They don't amount to a row of beans, and I'm not at all intimidated by them," said Bercow, who announced last month he would stand down as Speaker by October 31 -- the day Britain is set to leave the EU.

Bercow, whose cries of, "Order, Order!" have gone viral amid months of Brexit chaos, has at times been a divisive figure in the Speaker's chair. His supporters have hailed him as an ally to backbench MPs, while his detractors have criticized him for allowing opponents of a no-deal Brexit to take control of business in the House of Commons, against the wishes of the government.

Dismissing criticisms from hardline Brexiteer MPs, who claim that he is biased against them, Bercow said: "I wouldn't say I'm a remainer enabler. I would say that I'm an enabler of all colleagues across the House who want to express their different points of view."

He insisted that he was fair-minded as Speaker, suggesting instead that Brexiteers were bad losers. "If you are losing the match, it's quite bad form to blame the referee," he said. "I thought the Brexiteers were in favor of taking back control of Parliament being in the driving seat? Well, they can't have it both ways," Bercow said.

Bercow said that the passage of anti-no-deal legislation -- which Johnson is fond of describing as the "surrender bill" -- is "the most logical consequence if a deal isn't agreed" and Parliament doesn't want to sanction the UK crashing out of the EU without a deal.

But he dismissed speculation that he could take the helm of a caretaker government if Johnson was forced to resign in coming weeks. "I don't think anybody would seriously think that that is the likely resolution of this matter," he said.

"If you are saying, am I sitting by my phone and thinking people are going to come to me and say John! Rescue us from the imbroglio! The answer is I expect nothing of the sort, and I'm looking for nothing of the sort, and I am in fact very much looking forward instead to leaving the speakership and Parliament within a matter of weeks from now."

Asked about the bitter divisions within Parliament, Bercow pointed to the fact that the House of Commons was elected to represent the country -- and that, at pubs and dinner tables around the UK, people were arguing over the Brexit issue just as much as MPs.

"I would very politely point out that this Parliament, which some ministers have been so quick to rubbish, was elected almost a year after the EU referendum," Bercow said.

"Now that is not to invalidate the EU referendum. This Parliament was elected more recently, and it reflects the divisions which exist in the country as a whole. So if people say, 'Oh, well, Parliament can't come to a view, Parliament's divided, Parliament's constantly arguing,' well, that's true of a country as a whole. But the role of a parliamentary democracy is incredibly important."

Bercow admitted that he has never known a Parliament as rancorous, describing the bitter debates over Brexit as among the worst he's seen in his career. That rancor peaked late last month when Johnson suggested that the best way to honor the legacy of murdered Labour MP Jo Cox was to "get Brexit done."

His comments drew an outcry from across the political spectrum, with Corbyn accusing the Prime Minister of language that was "indistinguishable from the far right."

At the time, Bercow pleaded for more reasoned debate , telling MPs that the commons had grown "toxic."

Shown a clip of an address he made to Parliament in 2016 after Cox's murder, Bercow grew visibly emotional. "We've got to try to honor her and advance her legacy," Bercow said, urging lawmakers to "disagree agreeably" and work "across the aisle," as she so often did.