As the final live debate of the Conservative leadership contest came to an end on Monday evening, the two men competing to be Britain’s next Prime Minister turned to one another, smiled, and bantered out of range of the room’s microphones.

Officially rivals, Boris Johnson and Jeremy Hunt both seemed to recognize that the race to be the U.K.’s next Prime Minister had already been decided, even though the winner would not be announced until July 23. Johnson was the clear favorite among the Conservative Party members who will decide the contest when the starting gun was fired last month, and remains ahead coming out of the race.

By the end of Monday evening, the pair were joking with the audience, and one another. “I don’t think [the debate] changed very much from what we already knew,” Johnson told TIME moments after the event.

But while the leadership contest appears to be a foregone conclusion, its final weeks have been bruising for Johnson. Newspapers scrutinizing his past as a journalist and author have unearthed examples of past Islamophobic comments. They haven’t had to look far; in 2018, he likened Muslim women who wear the full-face veil to “bank robbers” and “letterboxes,” comments that prompted the Muslim Council of Britain to accuse him of pandering to the far-right. On Monday, the Guardian published excerpts from a 2006 book in which he had written that Islam “inherently inhibits the path to progress and freedom.”

And when, on Monday evening, Johnson was quizzed over a column in which he referred to gay men as “tank-topped bumboys,” he dismissed concerns over his past remarks. “If you’re going to excavate and disinter every single quotation from the millions of words I have written, you can of course try to twist things one way or the other,” he said.

That explanation might placate the roughly 124,000 Conservative Party members who have a vote in the ongoing leadership contest, but the wider public are yet to be convinced he is the right man for this job. In a May poll by YouGov, just 28% of U.K. adults surveyed said he would make a good Prime Minister, compared to 54% who said he would do badly.

That should be a cause for concern even if he does become leader. His ruling Conservative party has a parliamentary majority of just three votes, which could shrink even further if lawmakers abandon ship over his approach to Brexit. If that ends up forcing a general election later this year, Johnson would have to convince the entire electorate to put their trust in him as a leader.

So far, the contest to be Britain’s next Prime Minister has been fought on what appeals to Conservative party members, rather than the public as a whole. When Johnson eventually does go to the country, his past comments may come back to haunt him once again.

Write to Billy Perrigo at billy.perrigo@time.com.