Julian Smith, the chief whip, will survive in his post, despite two days of fevered speculation about his future. Several reports have suggested he told Conservative MPs to deliberately break their pairing arrangements for crunch votes on the Trade Bill on Tuesday evening, but Theresa May is sticking to the official line: that Tory chairman Brandon Lewis broke his pair with Lib Dem Jo Swinson and voted after an "honest mistake" by Smith.

For now, his job is safe. The fallout from the Swinson incident was always going to be survivable as long as no named Conservative MP broke cover and accused him directly, which they did not. But this unedifying episode is another blot on an already messy copybook. His standing among Tories both inside and outside of the Commons has been severely eroded (ConservativeHome has called for him to go). A "these things happen in the heat of battle" defence has limited purchase but politically, he is damaged goods.

The way this week has unfolded has contributed to a sense that things are not going well for Smith. This week's votes were particularly difficult, as the pairing brouhaha highlights. That he failed his first major test as chief whip in December – ensuring the EU Withdrawal Bill passed without defeat for the government – has not been forgotten either. Granted, managing this parliament would not have been easy for any chief whip. But there is nonetheless a belief Smith is singularly ill-equipped to handle it.

One issue is Smith's status as a whips' office lifer. He has never served in any other ministerial role, nor as a full cabinet member, which makes the job of negotiating with senior backbenchers more difficult (Gavin Williamson had at least been parliamentary private secretary to David Cameron). An abrasive personal style exacerbates this problem. "He is particularly poor at human relations," one senior Tory MP says.

As deputy to Williamson, Smith played the "bad cop" and continues to employ that threats-based approach as chief (moves to "crap offices", withdrawal of party funding). For some, however, he is nowhere near a weighty enough figure to make it work and its inherent unpleasantness corrodes the trust and the loyalty it is supposed to induce. His predecessor's style – which he memorably described as a "sharpened carrot" approach – is remembered by MPs as fundamentally rewards-based, despite his Machiavellian image.

Even if his days are numbers, as colleagues increasingly predict, there is the risk that the alternative is worse. One backbencher plaintively reflects: "Who could replace him and who I would want to replace him are two different questions." The "could" pool is tiny, and mostly occupied by people who May would not pick or cannot move from their current posts. And the warring Tory factions want – and, crucially, respect – different operators and qualities. It may be that this is a problem that simply cannot be solved while parliament and party are constituted as currently.