Every time I see Peter Lilley in the House of Commons I want to stop him and say: "Mr Lilley, the lost leader of the Conservative Party!" He came within moments of becoming leader of the opposition in 1997. A Cabinet minister under Margaret Thatcher and John Major, he stood against Michael Howard and Kenneth Clarke for the leadership after the Blair landslide. Had the contest proceeded thus, he would probably have won. Howard was winded by an unpleasant personal attack – "Something of the night about him" – by Ann Widdecombe, who had been a junior minister of his at the Home Office. The contest would probably have ended in a run-off between Lilley and Clarke. The fourth candidate, John Redwood, was never likely to win.

That was until a fifth candidate entered the contest: William Hague, the 36-year-old former Welsh Secretary, who had pledged his support to Howard. Thus it was Hague who emerged as the candidate of the Tory mainstream against Clarke.

The point of this ancient history being that Kenneth Clarke was not just a leading Conservative for more than a decade but that he was the leading Conservative against whom other candidates for the top job defined themselves. Two biographies of him were published in 1994, including an excellent one by Andy McSmith, my colleague on The Independent, in anticipation of his taking over from John Major as prime minister. But Major turned out to be stubborn, and Clarke refused to compromise on his pro-European beliefs. So Clarke became the candidate against whom other candidates ran. Hague beat him in 1997; even Iain Duncan Smith beat him in 2001. Clarke ran again in 2005, but his good-bloke name recognition had faded so much that he did not even make the last two.

Yet he had enough of that heavy stardust known as gravitas for David Cameron to recall him to the front bench in 2009, as by then the Conservatives needed someone in the shadow cabinet who had done more than carry someone's bags. He didn't really fit in. He is not just a pro-European, he is an older, gentler kind of Tory. He is a one-nation centrist and a liberal in a deep, 1960s meaning of the word. Cameron may have called himself a liberal conservative when he was paddling for Lib Dem votes at the election. The Tories may even have converted briefly to a strident and rather selective "liberalism" on peripheral questions such as CCTV cameras, DNA samples and, in David Davis's case, fighting by-elections to preserve Magna Carta, but Clarke was of a different kind.

In the early days of the Coalition government, Cameron used to joke, with a hint of bitterness, that dealing with the Liberal Democrats was nothing compared with having to argue with Clarke as Justice Secretary. So open were such disagreements that Nick Clegg, the Deputy Prime Minister, boasted to his annual conference last month: "We have five ministers in the Cabinet – well, six if you include Ken Clarke."

Then there was last week's dispute with the Home Secretary over immigration, which resembled a running brawl. The streetfighter in Clarke was roused, just as it was when Margaret Thatcher said at a conference fringe meeting in 1999: "In my lifetime all our problems have come from mainland Europe." When this was put to Clarke, he was disbelieving: " All our problems? Well, it's a point of view, isn't it?"

He was just as dismissive about Theresa May's story about the Bolivian overstayer allowed to remain here because he had a pet cat. The story was not untrue – the cat was mentioned by the judge in the initial case although the eventual ruling was on different grounds – but it was a foolish distraction from the serious proposal that May was making, which is that "respect for family life" under article eight of the Human Rights Act should not override immigration law. I do not know how practical it is, but her plan is not an unreasonable one. The public supports it overwhelmingly, which is why Cameron was so quick to take her side against Clarke, who was depicted by the liberal press last week as a cross between Martin Luther King and Wat Tyler.

I admire Clarke hugely. I think it is healthy for the Government that he goes on Newsnight and talks about prison as "an incredibly expensive way of accommodating people for a time, achieving absolutely nothing", although as a Blairite I don't share his insouciance about the Human Rights Act. It is a menace and an invitation to judge-made law. But reforming the Act is going to be difficult, and it will help to have someone of Clarke's experience and instincts to get it right.

Surprisingly, though, I do not think that it will be Clarke the liberal who causes Cameron the most trouble in the end. The other thing to remember about him is that he was a successful Chancellor of the Exchequer from 1993 to 1997. Indeed, it was he who laid the foundations of New Labour's golden economic age. His judgement was good. He faced down Eddie George, the then governor of the Bank of England, who wanted to put up interest rates, and was proved right. He was good at the politics too, planning for a spending freeze after 1997, even though he sounded cavalier when he later admitted he would not have kept to those limits.

But his common sense is dangerous to his successor, George Osborne. Remember that after the collapse of Lehman Bros in 2008, Clarke, believing a VAT cut would stimulate the economy, said: "There's no point in being ultra-orthodox." Alistair Darling then cut VAT, Osborne opposed it, and Clarke was invited to keep quiet. Darling is now thought to have got it right and Osborne wrong.