What is wrong with us? The United Kingdom should be one of the most impressive democracies on Earth. We have incisive and apparently incorruptible judges; an undeferential, boisterous and intelligent media; and an extraordinary culture of voluntary activity – supplemented by charitable superpowers such as Oxfam and the Royal Society for the Protection of Birds. Our civil servants, diplomats and soldiers are thoughtful and skilful. Our parliament is more diverse than at any time in its history. Our economy is stable. Only the US can rival our universities; and only the US and China, our success in science. London remains one of the greatest cities on Earth. Our citizens have never seemed so healthy or better educated. So why does it suddenly seem to be so difficult to make what once, at least, seemed obvious, sensible arguments from the centre ground?

Three weeks ago, I found myself in the final five for the leadership of the Conservative Party. I went into a BBC debate against the other four candidates – all of whom were promising unfunded tax cuts and increases in public spending. They also rejected the current European Union withdrawal agreement and insisted on retaining no-deal as a threat against the EU. All the others claimed to be able to get a new deal out of Brussels by the end of October (or in one case, the end of the year). All insisted that in the absence of such a deal, they would want to leave in 2019 with no deal, and that they would be able to drive no deal through parliament.

How could I lose against such arguments? Most of the public and 90 per cent of my parliamentary colleagues agreed – or at least had recently agreed – that we could not get no deal through parliament. Very few people were comfortable with unfunded tax or spending promises. Or with a no-deal Brexit. Or with suspending parliament. Nobody seriously believed that Brussels would offer an entirely new deal by October (even Nigel Farage agreed with me on that).

But I failed to win any of these arguments, and within 24 hours I was knocked out of the contest. There have been explanations for my failure. One, made by the pollster John Curtice, is that public opinion simply makes a “centrist” position such as mine impossibly quixotic. Public opinion was traditionally – in the time of Tony Blair and David Cameron – a bell curve with all the votes located in the centre ground. Now the bell shape has collapsed, like an unstable soufflé, into a U-shape, leaving voters only on the extremes.

Forty per cent of voters want to ignore the referendum result and remain in the European Union; and 40 per cent of the population – and around 80 per cent of Conservative members – apparently favour no deal if the alternative is remain. There was almost no constituency for someone trying to argue for a moderate and pragmatic Brexit among the public – and therefore there could not be among practical MPs.

I favour a different explanation. Which was that I had forgotten all the lessons of the many walks I had done around the country in the previous weeks – from Derry to Derby, from Edinburgh to Peterborough – and had tried simply to rehearse what I saw as the facts. I talked about the impact of unfunded tax cuts and spending pledges on our fiscal position. I tried and failed to explain in a few seconds how higher tariffs following a no-deal Brexit would lead to inflation, pressure on incomes, interest rate rises, and ultimately negative impacts on GDP.

In other words, as some of my friends argued when they were being more polite, I was “off my game”.

The lesson is that it is possible to change minds, and defeat extremist positions; but not by explaining tariff levels. If 72 per cent of voters are dissatisfied with the UK democratic system – half believing that the government doesn’t care about them, and more than half saying that “Britain needs a strong leader willing to break the rules” – you cannot expect to win simply on technocratic arguments. But nor should you feel forced to respond with nonsense and fairy tales. What I had seen, walking around the country, is that democratic life is neither about echoing and deepening pre-existing prejudices, nor only about communicating economics.

***

To support a sensible, pragmatic position, you have to begin by rediscovering a sense of anger and shame. And acknowledge how unforgiveable and appalling many things are in modern Britain. You need to see the line of broken windows in cell after cell in a Liverpool prison, and see the blood on the floor in one in Birmingham, to feel the right kind of shame at the state of our prisons. You need to get out to Poplar in east London to see the white tent in the rough grass between the buildings of a housing estate, pitched over the body of a dead man; the air ambulance, rotor blades still spinning, waiting to take him away; and listen to the older man explain what it was like to walk out of his mosque and see someone lying on the ground, bleeding to death from a knife wound.

Second, you need to be very much in a hurry to fix things. You cannot accept that it is impossible to reduce violence in prisons in a year. Or that the fast train connection between Leeds and Manchester will not begin to be built until 2035. Or that Britain should have slower broadband speeds than Madagascar. Or that almost every young man around that Poplar estate is carrying a knife. Nor should you accept smaller injustices – such as the idea that a constituent visiting a dying parent should receive a parking fine in the hospital car park.

Third, you need to be ambitious. You should aim to plant not 11 million trees but 110 million trees; build not a million homes in five years, but two million and build them with government money, and make them beautiful.

And if there is a single extreme disgrace, such as the great unfinished revolution of adult social care – left for decades after the creation of the NHS – then sort it out. It is not defensible that the frail elderly should only get 15-minute daily care visits in their homes – hardly long enough to wash them, let alone talk to them. So, reach boldly across parties, and agree on how to finance a proper system of care.

All of this – seeing the horrors, feeling the shame, getting on with it at scale – allows you occasionally to do something more fundamental, which is to be truthful about your own failings, and the obstacles that stand in the way of getting anything significant done. And all of this – from shame to truth – helps you to believe in what you are doing again.

Perhaps because the centre ground had seemed so inevitable and successful for so long, little emotional or moral energy was needed to make the case for moderation. Instead the arguments had long been surrendered to policy think tanks and civil servants who produced intelligent papers, in a minor key – about learning lessons from Scandinavia, or recommending better use of technology, or more efficiency. The only “value” that had been discussed for two decades was value for money. Patriotism, liberty, courage, and the will of the people were ideas that were abandoned – left on a dusty shelf – to be picked up again by the more divisive politicians.

It was all too easy for the extremists to use punchy language to disguise their vacuous fantasies, making the impractical seem practical, the negative positive, and the airy-fairy down to earth. Take “no-deal Brexit”, for example. It is a phrase that sounds like a thing, a fine Anglo-Saxon fact, not an abstraction like “a temporary backstop”, but it is in fact the opposite of a thing. No deal is an absence that pretends to be a presence; the negation of a deal that pretends to be a type of deal.

It is a phrase that floats recklessly free from any connection to any particular person in a local place, or at a particular time. It cannot ever tell, say, Chris Harrison, a sheep farmer in Alston, Cumbria, whether he would have to pay European tariffs of 40 per cent on his Swaledale lambs; or whether Brazilian beef would come in import-free. But it is vague enough to allow its proponents to claim it is democratic despite the majority of the population being against it, and patriotic even though it is against the best traditions and interests of our country.

Like almost every other argument made today – even in the mainstream of the Labour and Conservative parties – it builds on the fantasy of the victim, that you would be better off on your own, if only you could rid yourself of other classes, other groups and other nations. It shares the instinct that made some Scottish nationalists feel that the many challenges of a modern nation could be solved by simply getting rid of England. And these desires to turn away from relationships seemed an aspect of a deeper desire to turn away from reality – from the questions of who or what we really are today as a people, what we share, and who we can realistically be in the future.

***

I’ve been struggling to communicate my sense that the centre ground should not be simply a midpoint between these empty abstractions – some grey smudge between the blacks and whites of the extreme; that it has a reality, a connective power and a maturity that is denied to the other positions. It is something that exists on a different dimension – of reality as opposed to fairy tale. And because it is grounded in reality, it can be sustained and successful in a way populism cannot, because it is founded in truth.

But the centre should also not be a narrowing project that seeks to belittle the intuitions, the understanding or the motivations of people who currently favour a no-deal Brexit or Jeremy Corbyn.

Faced with a democracy that sometimes feels as though all the votes are gathered at two opposite ends of a stick, the centre ground must not be simply the midpoint of the stick, whose only merit is being as far away as possible from each extreme. It should instead involve a project of bending the stick and connecting both ends with a string to make a bow. The centre should then be at the midpoint of the string, the point of greatest potential energy – which comes not from excluding the two extremes, nor from linking them loosely together, but instead from harnessing the tension of two opposing forces.

Our country has entered a midlife crisis. The answer cannot be to try to lurch back to an adolescent fantasy of being saved by superheroes, but instead to move forward into maturity. Such a maturity begins with a recognition of our shame, our failures, and our successes in surviving and battling the constraints of the world and time. A maturity that turns from comforting and ambiguous abstractions to face, unflinchingly, reality. And to say, like Samuel Beckett’s Krapp, reflecting on whether he missed the excitement of his youth, “not with the fire that’s in me now”.

Rory Stewart MP is Secretary of State for International Development