Can anything destroy the Teflon Speaker? MPs on parliament's standards committee have voted against launching an inquiry into bullying allegations against John Bercow.

The decision, passed by a margin of three to two, means the Speaker will not face a formal probe into claims that he mistreated Angus Sinclair and Kate Emms, two of his former private secretaries. He denies all allegations of bullying, but his near-omnipotent status within parliament's arcane and opaque structures means that an investigation by the standards commissioner was the only real recourse for those who wished to see him held to account.

Now, in the absence of fresh controversy and without independent scrutiny, it appears that Bercow will hold on. His critics - both in and outside of government - are deeply unhappy. The latter group are increasingly exasperated by quirks and deficiencies wired into the limited set of procedures that might have been used to investigate Bercow.

Rules dictate that the committee must vote on any proposed investigation by the commissioner that is more than seven years old, which meant the decision ultimately lay with a handful of MPs – many of whom, as I have written before, have a stake in protecting a Bercow speakership.

While Labour's Bridget Phillipson and Conservative Gary Streeter voted for an investigation, Labour's Kate Green and Tories Christopher Chope and John Stevenson did not. There are lay members on the committee, but they did not have a vote. Kevin Barron, the Labour MP who chairs the committee, did not waste any time in pointing this out in the wake of last night's vote. His complaint to that effect was telling. The appetite to oust Bercow exists at the highest level within parliament, but the constitutional means do not.

The decision highlights another challenge too. If Bercow is to go, then a critical mass of his remaining admirers in the Commons – of which there are more than the relentless slew of terrible PR suggests – will have to make the calculation that the allegations are bad enough to outweigh the qualities in the Speaker they admire or benefit from. That Chope and Green backed the Speaker suggests we are some way off that point yet.

Traditional Tory Brexiteers, such as the former, have long been grateful to Bercow for giving their once-fringe cause parliamentary time before the EU referendum, while the latter reflects a tendency among Labour MPs to value Bercow's modernising approach.

In recent weeks, the Speaker has ostentatiously reminded his wavering supporters of his qualities: highlighting the hundreds of urgent questions he has granted during his tenure and chastising the government for eating into opposition time.

Though unease is growing on the Labour benches, and the ambient dislike for the Speaker is as strong as ever among Tories, last night's vote highlights challenges that, though by no means insurmountable, will take more time and energy than most are willing to expend. Add to this the lack of procedural avenues left to pursue, and the picture is even bleaker for the anti-Bercow cause.

Unless new, fresher evidence of alleged misconduct surfaces and changes either of those dynamics, it is hard to see the Speaker doing anything but limping on towards a departure date of his own choosing.