Boris Johnson’s return to parliament is not going according to plan. With the Conservative party leadership apparently out of reach for the next few years, our most celebrated political princeling has been reduced to the hugger-mugger life of a backbencher, sharing lobby fodder status and green leather benches with MPs he had thought himself destined to rule from afar.

No wonder Johnson looks uncomfortable. There is nothing he dreads more than people getting up close and personal. And indeed now that they have done so many of his onetime potential supporters do not like what they see. In fact, this week came suggestions in the Sun (in the past a newspaper supportive of Johnson) that a majority of the other “new” Tory backbenchers have decided, on closer inspection, that their putative electoral hero is not as impressive or solid as they had been led to believe.

They don’t like the way he turns up to political cabinet in his bicycle helmet and apparently without having read his briefs, or how he makes long-winded attempts at humour on the floor of the house, or the fact that on a one-to-one basis he is often aloof and unempathic. Johnson is not really, they now apparently believe, “serious” at all. He just wants to be the star.

It is a familiar assessment to many who have worked closely with him. Johnson’s performance in City Hall should have provided clues that he may not be cut out to be a top-flight politician. Of course, the fact that he has been elected mayor twice in a city that has traditionally favoured Labour is a significant achievement – and testament to his extraordinary vote-catching powers. His huge popularity has until now overshadowed negative reports about his regime in London.

But his election to Westminster has swivelled the spotlight on to exactly what he has achieved beyond the ballot box. Suddenly, his track record matters. And it is not good for him that non-political officials at City Hall complain about delayed decisions, obsession with vanity projects and a lack of direction. The fact that Johnson also has a weekly newspaper column, writes and heavily promotes his own books, as well as now being MP for Uxbridge, has hardly helped keep the place humming with vision and energy.

Sources at City Hall say that it is down to Johnson’s chief of staff, Sir Eddie Lister, that anything gets done at all in a city beset with problems, from pollution to housing. London may have prospered overall on Johnson’s watch but his growing number of critics fear that not enough has been done to safeguard its future.

Good political leaders also need to be convincing under pressure. Johnson’s belated and seemingly smirking reaction to the riots of 2011 was anything but. “Boris failed as mayor during the riots,” a source close to David Cameron told me back then. “He failed to be a leader.”

The purchase of three cut-price German water cannon could have diverted attention from these failings. Instead the deal ended in humiliation when the home secretary, Theresa May, refused to authorise their use last week. Boris has not dealt well with the fallout. Normally the master of media attention, he tried to duck difficult questions in one TV interview on why he had wasted taxpayers’ money by pleading that he had to dash off to another. But then, Johnson has never coped with scrutiny. Press him on his fitness to govern, his spurious statistics, his cavalier treatment of taxpayers’ money or his failure to protect London’s cyclists, and he is more than likely to respond with an evasion or an insult than a convincing argument.

Members of the London Assembly have been regularly called stupid or “care in the community” by him simply for trying to hold him to account. There is little in his conduct at mayor’s question time sessions to suggest he is properly equipped for PMQs.

They don't like the way he turns up for political cabinet in his bicycle helmet apparently without having read his brief

Johnson has long had plenty of critics. Those who examine his track record objectively tend to come away distinctly unimpressed, although – bar the court of appeal, which two years ago said that his “recklessness” in committing “extramarital adulterous liaisons” specifically raised questions about his fitness for public office – they have largely remained silent.

The election result, which destroyed his immediate hopes of the Tory leadership, has simply given his critics a chance to speak up. Johnson’s lack of loyalty to others during the years of sunshine has left him with few friends now that it has begun to rain.