Last week, Ken Clarke left the cabinet. Here, in a candid and revealing interview, he recalls the rows with Thatcher and Cameron, his fears for the party – and his rule for staying calm

As the day of the reshuffle grew closer, most ministers sweated in their offices, waiting for the call from No 10 that would reveal their fate. Not so the insouciant Ken Clarke: he took himself off to Nottinghamshire to watch the cricket.

"I went Wednesday, Thursday and Friday," he chortles. "Even my diary secretary knew what I was up to. I told her it was a vital constituency engagement, but she knew perfectly well that Trent Bridge is in my constituency."

He had already concluded it was time to draw stumps after what he calls his "embarrassingly long" career on the front line of British politics. He expresses surprise that David Cameron didn't asked him to retire from the cabinet two years earlier. "By the time you are at my very advanced age, you really ought to start thinking when you're going to retreat with some dignity."

It brings to a close an impressive innings at the cabinet crease: David Cameron was three and George Osborne hadn't been born when Clarke arrived on the scene. The son of a jeweller and watchmaker, he entered parliament as a 29-year-old lawyer from the Midlands in the election of 1970. He has served in the governments of every Tory leader since: Ted Heath, Margaret Thatcher and John Major before returning for an Indian summer to bring some heft and experience to Cameron's top table.

And it is a bit misleading to suggest that he is retiring; was Ken Clarke ever retiring? It is only the cabinet he is leaving, not politics. He is determined to stay on as an MP, for he remains "addicted to politics" and "the mad mayhem of it all". For this veteran and implacably pro-European Tory, there is also one final battle to be fought: to prevent "the rightwing headbangers" in his party from wrenching Britain out of the European Union.

Education, health, home office, chancellor: few people have done so many big jobs. His last department was justice, where he was berthed for the first two years of the coalition's life. He now thinks that "turned out to be a slight mistake" and the reasons he gives are revealing about Cameron's style of government. "I was very pleased to go to justice … and I had lots of views on it, but unfortunately my views didn't coincide with No 10's."

His was a liberal-minded approach – Nick Clegg joked that he counted Clarke as the sixth Liberal Democrat member of the cabinet – and it collided with a Cameron regime at Downing Street that was "blisteringly rightwing on law and order". He explains: "David and I didn't fall out exactly. There was a constant friction. The worst of it was, some of his aides were briefing that I was going to be sacked all the time."

It got so bad, he reveals, that the prime minister's people tried to gag him – not an endeavour ever likely to succeed with a politician famous for telling it as he sees it. "I had a great row with them when they told the producer of Question Time that I was ill and they were able to provide a replacement. It never occurred to them that I could ring up the producer. She said: 'I'm told you are ill.' After that, I got even more freelance."

His off-message appearances in the media made Lynton Crosby, the prime minister's pollster, furious. He was delighted to read "the story about Lynton cursing one of my appearances on the Today programme, which I had done deliberately just because it wasn't on the grid".

When he reflects on his time at the top of British government, what is the most important thing he's learned? What would be Clarke's rule of politics? "I think my rule is: it's never as bad as you think. At any given moment, it can seem remarkably chaotic. The most odd events occur. It's never the smooth, organised, strategic picture that you think it's going to be. The thing not to sink into is the 'flap of the day' approach to running things."

He recalls one of his worst moments – when the Daily Mirror leaked his entire budget speech the day before he was due to deliver it. "Whereupon flapdoodle broke out. People were saying: 'We've got to cancel the budget.' I was surrounded by people running around in ever-diminishing circles. So I said I was going for a curry and I hoped we'd have a lawyer when I got back to give advice on whether we could stop the leak. And we did. We got one of those midnight injunctions."

The "absurdities" of politics are a recurrent theme of our conversation. As reshuffled ministers take up their new posts, he offers this warning. "You arrive at a department you hadn't thought you were going to get sent to. You spend the first six months faffing about. After about six months, you've decided you know exactly what to do. After about two years, you realise you're screwing this up, you didn't get it right, but now you really understand it. And then the bloody phone rings and you're moved to another department and you go back to square one."

The best jobs he ever had, he says, were as health secretary and chancellor – "both fascinating stuff" where he had a long stint that gave him the chance to "deliver my own agenda". His first challenge at health was heading off Thatcher, who "wanted to go to the American system", he reveals. "I had ferocious rows with her about it. She wanted compulsory insurance, with the state paying the premiums for the less well-off. I thought that was a disaster. The American system is hopeless … dreadful."

He prevailed on her to take a different route by introducing more competition into the NHS. It became known – in a phrase he didn't like – as "the internal market". Ever since then, successive governments have pushed in broadly the same direction.

His career peak was as chancellor, a job in which "you cover the waterfront of the whole damned government". He picked up the pieces in the wake of Britain's shattering exit from the European exchange rate mechanism on Black Wednesday, and presided over a resumption of growth that lasted until the great recession.

His regret is that the 1992-97 Conservative administration was such a nightmare of fratricidal infighting, mainly over Europe. "The Major government was bad because it was permanently a lunatic civil war and the government was scarcely functional. The government was falling apart."

Three times he ran for the Tory leadership; three times his party rebuffed him. "I obviously regret it." He takes consolation in the thought that "the club of prime ministers we nearly had is quite a respectable collection of people. [Roy] Jenkins and [Denis] Healey and [Geoffrey] Howe and me. Quite a distinguished lot who might have been prime minister. My problem was I was too pro-European."

Indeed it was. There have been two tectonic shifts in his party over the span of his career, both of them unhappy developments for a man always faithful to the one-nation, pro-European tradition of Toryism.

The first major change is that the Conservative party has lost its previous talent for dominating British politics by assembling coalitions of voters. What was once the most successful party in the democratic world has not secured more than 40% of the vote, or a parliamentary majority, since 1992. Does he see any evidence that his party can rediscover how to win? "Not enough," he says. "It is one of the lines I often use with my colleagues: I belong to a Conservative party that used to be able to win elections."

Reminding Cameron that he failed to win in 2010 must go down a treat. "No," laughs Clarke. "Not one of my popular sayings with David." He sounds very doubtful that his party can win an overall parliamentary majority next May. He is more confident in predicting that "we'll be the biggest party".

The key terrain will be the economy. He is too large a character, and too candid a one, to talk about it in bullish soundbites. The recovery is "a work in progress" because "it's not firmly enough rooted on a proper balance between manufacturing and a wide range of services and financial services". The banking crisis "isn't resolved"; the international outlook is "fragile". There is "a long, long way to go" in terms of skills and investment to create "a modern, competitive economy capable of taking on the Chinese, Brazilians and everyone else."

On this, at least, he agrees with Ed Miliband. There's an echo of the Labour leader when he says: "We don't want to be a low-wage, low-productivity, long-hours economy." He also sees danger in repeating the "ludicrous cycle of ridiculous housing booms followed by housing crashes".

For all that, he believes the economy will be the Tories' trump card over Labour because "there's absolutely no doubt the public usually prefer a Conservative government when they're worried about the economy."

Painting Miliband as unfit to be prime minister is the plan among Tory strategists. "I'm not squeamish about my trade," says Clarke, but he expresses distaste for ad hominem attacks. "This is a generational thing. One of the ways in which politics has changed which I don't find very desirable is … the celebrity culture. The system gets more and more presidential." A result is "the personal attacks on individuals". Mind you, the pugnacious Clarke has never pulled his own punches. "That's right, but not by going out making offensive attacks on my opponents. I'm old-fashioned. I always got on with everybody who shadowed me. I even got on with Gordon Brown."

What would happen in the event that Cameron crawled over the finishing line as prime minister with no majority, or a very slim one? He surely fears a rerun of the John Major experience: Nightmare on Downing Street II.

Clarke doesn't argue. Cameron "would have sleepless nights at the thought of leading a party that had a majority of half a dozen … The trouble with having a tiny majority … is you're totally at the behest of difficult colleagues who have tremendous clout."

The other major change during his career is the ascendancy of Euroscepticism and Europhobia in his party and the dwindling of Conservative pro-Europeanism to the point where it is an endangered species. He says there are some pro-Europeans among younger Tory MPs, but agrees that "they keep their heads down" for fear of the phobic newspapers and the "rightwing headbangers" who dominate Tory debate. "I'm not saying every Eurosceptic is unintelligent, but there are some pretty daft ones given very great prominence."

He acknowledges that in today's Tory party, "everybody feels obliged to say they're Eurosceptic". He has some fun at the expense of Lord Hill, Cameron's candidate for Britain's next European commissioner. Clarke knows Hill very well. "Even Jonathan Hill seems to have said he's Eurosceptic. News to me. But it is necessary to say you're Eurosceptic, it would appear."

He laments that "the pro-European case is not being put by anybody. One of the things we're going to have to do over the next two or three years, pro-Europeans of all parties, is sort out how we start putting a sensible pro-European case."

One of the reasons for the "neurotic debate", he adds, is that "there's no reporting of European politics here. None whatever. The politics of continental countries is less reported than that of the state of New Jersey."

A persistent flaw in Tory prime ministers is to forget that they are not the only leaders with domestic audiences to satisfy; similar pressures apply to their continental counterparts. "I had the conversation with David Cameron … and with John Major. I remember saying to John Major that trying to persuade other leaders that the most important thing is to get a good writeup in the Daily Telegraph is difficult, because it's not the most important thing as far as any of them are concerned."

As for Cameron's attempt to block Jean-Claude Juncker as president of the commission: "I think it was the British press that gave Jean-Claude the job. Suddenly, the rightwing press were attacking this Luxembourger who has never caused a moment's excitement to anybody for the past 30 years.

"The idea that they [European leaders] could go back to their own countries and media and say we have to get rid of Juncker because the British media were going bananas about him – how on earth could you get away with that? It looks so pathetic and wet."

He doesn't sound optimistic about Cameron's chances of bridging the gap between what the rest of Europe might agree to in a renegotiation of terms and what much of the Conservative party will demand.

"Most Eurosceptics don't know what reform they want, and the hardline Eurosceptics are fond of demanding any reform which is completely inconsistent with membership of the union."

He is of a generation that doesn't like referendums in principle. He evidently thinks – even if he is too polite to say so – that Cameron was mad to concede one to appease his MPs. But he "can see the argument that until we have a referendum we are never going to put this to bed".

The stakes are high, he believes: nothing less than the country's standing in the world. "Being prime minister will be a less important job than it's ever been before if we leave the EU. You'd become a minor player in all kinds of situations where the British are entitled to have a reasonably leading role." If there is a referendum, does he see that as the last, great battle of his career? "I will admit it. It's one of the principal reasons why I haven't lost my enthusiasm. To my amazement, I am still conducting the same arguments that I was when I was a student politician. I'd like to see it through to its conclusion – if such a thing is possible."

He may be retiring to the back benches. But the veteran batsman of one-nation, pro-European Toryism is most definitely not retiring from the fray.