Since he entered parliament in 1970, Kenneth Clarke has served under eight Conservative leaders, from Edward Heath to Theresa May. He has also stood three times, in 1997, 2001 and 2005, to be Tory leader himself during difficult periods for his party. But throughout it all he has never known a political crisis remotely like the present.

Now aged 78, he still describes himself as “a natural optimist”. But a combination of Brexit and the Tory leadership contest are testing his positive thinking to its limits.

“As someone who has seen a few leadership elections in my time, this one’s quite different,” Clarke said in an interview with the Observer last week, within hours of the announcement that Boris Johnson had stormed into the lead in the first round. “This is a tragic farce of a crisis in which anger and protest are wide sentiments across the public scene. The Conservative party is in turmoil internally and deeply unpopular with the general public.”

As a lifelong europhile its hurts Clarke, who as father of the House of Commons is its most senior MP, to think that he will be retiring at the next election with what he calls this “extreme crisis” caused by Brexit overshadowing everything. He describes leaving the EU as a “crazy decision” – one which will leave in ruins much of what he has tried to do over the past five decades.

“My entire political career has been based on building up Britain’s political standing and economic prosperity through our membership of the EU and the European project,” he says. “Now it’s all come to an end and the political system and parliament is in such mayhem and incapable of dealing with the crisis that this referendum has provoked.”

He is also dismayed by the way the Tory party is going about choosing its new leader and the country’s next prime minister. “There’s an air of fantasy about just about every candidate’s campaign,” he says, with promises being made on Brexit that cannot be met. And the way the eventual decision will be handed over to the Tory membership, he finds simply “extraordinary”. “Most dictatorships have a better way of choosing their leader than this one.”

On Boris Johnson, Clarke is withering in a way that borders on contempt. He says he was a “disaster” as foreign secretary, and when asked if he could be equally terrible as prime minister, he answers in an instant. “Yes, he certainly has the potential to be. Unless he suddenly starts taking it seriously. I am not sure he knows what he would do to get us through the crisis.

“With the exception of Rory Stewart [the most pro-EU candidate, who Clarke is backing] they seem to be implying that they have some magic key to changing reality and getting the withdrawal agreement re-opened, or getting some marvellous technology that can check every lorry crossing the Irish border without having to have any customs officers there. The idea that we just leave on 31 October with no deal is utterly farcical.”

That the Tory party’s most senior MP and highly respected elder statesman, who has been chancellor of the exchequer, home secretary, health secretary, education secretary and lord chancellor, can describe the probable next Conservative prime minister in such a way, and the entire political system as in the grip of a self-inflicted crisis, is remarkable. And it is not just an isolated view from one veteran politician.

Many at Westminster share Clarke’s analysis that politics is utterly broken, and that what is currently happening, particularly with the Tory party leadership, could well make things worse. Most MPs, though, are far more discreet about saying so, as their careers lie ahead of them. Clarke, by contrast, no longer has ambitions to fulfil nor does he feel he has to answer to his local party. He says precisely what he thinks. And much of what he says resonates very widely with the public, as shown by the striking polling and focus group work conducted by Britain Thinks that we publish today.

Rory Stewart, who has the backing of Ken Clarke, launches his bid to become leader of the Conservative party and prime minister. Photograph: Leon Neal/Getty Images

The Britain Thinks examination of the nation’s confidence levels and views of its leaders uncovered a deep pessimism and extraordinary distrust of politicians.

Some 74% of voters say the British political system is not fit for purpose, with only 6% saying that UK politicians understand them, and 72% saying they don’t. Some 52% think that Johnson will be the next prime minister, but only 21% have faith that the next prime minister, whoever it may be, will be up to the job.

The polling found people felt more engaged with politics post-Brexit, but that extra engagement seems to have left them more frustrated than ever at their politicians’ failings.

“I feel pessimistic about the future simply because of the uncertainty, not knowing what’s to come and the lack of trust I have in all politicians and all parties,” said one focus group participant in Leicester. Increasingly, also, Britain’s inability to cope with its political crisis is being noticed abroad.

The latest cover of Time magazine is headlined “How Britain went Bonkers. The Brexit fiasco” with a picture of Theresa May and Tory leadership contenders, including Johnson, on top of a London bus grinning, as the bus sinks.

Scott Wightman, Britain’s outgoing senior diplomat in Singapore said in a valedictory note last week that the UK was now seen overseas as a country beset by division and oblivious to truth. The nation that Singaporeans “admired for stability, common sense, tolerance and realism grounded in fact, they see beset by division, obsessed with ideology, careless of the truth …” He added: “I fear many around the world share their view.”

Today John Kerr, a former UK ambassador to the EU and the author of article 50, says he is dismayed by the false promises being pushed out by the Tory leadership candidates, which will be treated with disdain by the EU if and when they have to try to deliver on them.

“What alarms me most about the current Conservative party leadership race is that fiction and fantasy are back, and harsh facts again forgotten,” Kerr says. “The unicorns are back, frolicking in the Tory forest.”

He says the claims by Johnson and others that the withdrawal agreement can be renegotiated “ignore the solemn undertakings given by the UK government in March that it will not seek to do so, as well as the EU’s repeated statements saying it will not do so”.

All this feeds the impression abroad that UK politicians are not operating in the real world any more. “Margaret Thatcher and John Major knew that bluff and bravado doesn’t work in Brussels. Not paying what we owe, and have promised to pay, wouldn’t change the game in our favour: it would end it. And, as the CBI and TUC rightly warn, leaving with no deal, and hence no transition period, would be catastrophic. “No deal isn’t backed by the country, or the Commons. So it really matters that an auction of promises to ultras doesn’t become determinant of our nation’s future. Facts matter.

The central promise of Johnson’s campaign is that he will take the UK out of the EU on 31 October “deal or no deal”, saying that not to do so would be a betrayal. The implication is that parliament will not be allowed to stand in the way of Brexit.

His supporters hope that if he wins among Tory members in the country, he could instantly revive morale and win back the many Brexit-supporting voters who have deserted to Nigel Farage’s Brexit party. Some believe he could then call a general election in early autumn, to gain his own mandate from the country to take the UK out of the EU on time.

But while Johnson looks on course to win the leadership of the Tory party, that is as much as it would be wise to predict. Honouring his promise to deliver Brexit will be no easy task. “If we get lured into thinking that Boris solves everything, we could be in for a very nasty surprise indeed,” said one former minister last week. “He could actually make this whole crisis much, much worse.”

If and when he enters No 10, Labour will almost certainly table a vote of no confidence in a Johnson government that tries to take the country out of the EU without a deal, against the majority will of MPs in parliament. Last Wednesday the Tory MP and former attorney general Dominic Grieve said he would be prepared to vote down any government if it tried to act in defiance of parliament’s will.

In an emotional statement in the Commons he told MPs that “the only way of stopping that prime minister [implementing a no-deal Brexit] would be to bring down that prime minister’s government... I will not hesitate to do that ... Even if it means my resigning the whip and leaving the party.”

Clarke told the Observer he, too, would feel bound to do the same and thinks other Conservatives could follow suit, though he does not know how many. Tory MPs bringing down a Tory government? Would that not elevate the political crisis to new levels?

“Well, I mean, if there’s no other way ... then you’ve got to bring that government down,” Clarke said. “You can’t have somebody saying ‘I’m going to be a dictatorial president for a month or two and fix everything despite parliamentary disapproval. If ... some idiot was sailing onto a no-deal Brexit, I’d decide politics had finally gone mad and I was not going to support this.”