Manchester, UK (CNN) Since becoming British Prime Minister in July, Boris Johnson has suffered political defeats virtually unheard of for any holder of the office.

Johnson's inner circle had not been particularly worried by any of them. Aides believed the setbacks played into his narrative that the British establishment was frustrating his Herculean efforts to "get Brexit done," as one of the Prime Minister's favorite slogans puts it.

Then, things took a more problematic turn.

This weekend, Johnson was accused by a respected London journalist, Charlotte Edwardes , of squeezing her thigh at a lunch in the late 1990s or early 2000s, when he was editor of the Spectator political magazine and she was a contributor.

Edwardes wrote in the Sunday Times of London that Johnson placed his hand high up her leg and had "enough inner flesh beneath his fingers" to make her "sit suddenly upright."

In her column, which appeared just as members of Johnson's Conservative Party were gathering for their annual conference, Edwardes went on to say that Johnson behaved in the same manner to a woman also present at the lunch.

The Prime Minister's official spokesman -- who usually confines hidmself to political matters -- denied the claims in a one-line statement. "This allegation is untrue," it reads. Confronted with the Edwardes allegations during a visit to a factory on Monday, Johnson was equally blunt. "No," he said, in reply to a question about whether the incident happened.

But Edwardes stood by her claims. "If the prime minister doesn't recollect the incident then clearly I have a better memory than he does," she posted on Twitter later Sunday

It has been widely reported in the British media that Johnson had more than one extra-marital affair -- claims that Johnson usually brushes off with a refusal to comment on personal matters. But before now, there was no suggestion that Johnson had ever imposed himself on women.

That's what makes the Edwardes allegation so toxic for him. Aside from the fact that no British Prime Minister in modern times has ever had to issue an on-the-record denial of groping, it wrecks the image that Johnson had carefully cultivated for himself -- rakish, but ultimately a decent chap. You might not want your daughter to marry him, but at least he's not a creep. So far, the strategy has worked -- his personal life has already been priced in by his supporters.

That might be about to change.

The groping allegation comes hot on the heels of another dangerous story for Johnson. On Friday, it emerged that the Great London Authority (GLA) had referred him to a watchdog body, the Independent Office for Police Conduct, to assess whether it is necessary to launch a criminal misconduct inquiry over claims surrounding his friendship with an American businesswoman, Jennifer Arcuri -- the pair met when Johnson was campaigning for his second term as mayor of London in 2012.

"Allegations have been brought to the attention of the Monitoring Officer that Boris Johnson maintained a friendship with Jennifer Arcuri and as a result of that friendship allowed Ms Arcuri to participate in trade missions and receive sponsorship monies in circumstances when she and her companies could not have expected otherwise to receive those benefits," the GLA said in a statement.

Arcuri and her company have not responded to CNN's request for comment. She told the Sunday Times, "any grants received and any trade mission I joined were purely in respect of my role as a legitimate businesswoman." Downing Street has declined to comment to CNN on the allegations, but Johnson said in BBC interview that everything was done "entirely in the proper way." Downing Street has yet to grant CNN an interview with Johnson.

If Johnson becomes the subject of a formal investigation, it could all get very serious. The offense of misconduct in public office is a serious crime in the UK.

On Sunday night, as the annual Conservative party conference got underway in Manchester, northern England, party members remained in high spirits. The latest allegations merely reinforce the Johnson-versus-the-establishment narrative, they think.

As he breezed through the crowded bar in the conference hotel on Sunday night, the Prime Minister was cheered by party delegates and supporters.

But multiple delegates to the Conservative party conference -- a mix of aides and Members of Parliament -- expressed nervousness to CNN. They are worried that Johnson is increasingly appealing to a smaller group of voters and are openly talking about the consequences if he's found guilty of misconduct.

And the political and legal problems facing the Prime Minister are also mounting. Aside from the Arcuri scandal, he lost a case in the Supreme Court last week, when 11 justices unanimously ruled that his suspension of Parliament was unlawful . And he might soon find himself up before the justices again, if, having failed to get a deal from the EU, he refuses to request a Brexit delay from Brussels as mandated by a law that came about after recent political defeats.

As of last week, the image of Johnson the Brexit hero, battling to tear down the Europhile establishment, was exactly what Downing Street wanted. This is exactly the sort of image they hope to project in the election campaign that pretty much everyone believes will come sooner or later.

But with legal cases and sleaze allegations piling up, Johnson is looking like a leader fast losing control. That might explain the scenes in Westminster last week, where Johnson engaged in some particularly nasty exchanges with opposition MPs in Parliament.

The problem Johnson faces is that lashing out only works if your subject feels intimidated. And right now, no matter how much he and his allies hit back at their critics, it feels like Johnson is the one being forced onto the back foot.