When David Cameron took over as Tory leader in 2005, he inherited a party that was downhearted after three electoral defeats. It looked and felt out of touch with the nation, symbolised by having just 17 female MPs and two from ethnic minorities backgrounds. He proclaimed the need for modernisation of both party and policies, pledging to move back to the centre and give the parliamentary group a makeover. This was resisted by his foes on the hard right, some of whom even attacked the proclaimed pursuit of environmental reform.

Cameron’s modernisation worked on one level. Look at Boris Johnson’s cabinet and it is heartening to see four cabinet ministers from ethnic minorities sitting round the table. He succeeds the second female prime minister, a politician who once rightly questioned if her party was really “at ease with 21st-century Britain”, while women hold several key posts in the new government. More than one in five Tory MPs are now female – still far too few but a big advance – along with 19 from ethnic minorities and the same number saying they are gay or lesbian.

Instead of rooting out the hard right, David Cameron ended up empowering them through a foolish referendum

Yet clearly these changes were cosmetic. The tragedy of the Tory party – and of Cameron’s catastrophic failure of modernisation – is that his capitulation to a fringe group of nativists has ended up leading to their takeover. Instead of rooting out the hard right, he ended up empowering them through a foolish referendum – and now they have moved the Tories into hideous terrain. Some call it a coup as Johnson’s cabal, spewing out hollow words of unity, drives out those with any doubts over no-deal exit from the European Union. Diversity has its limits in his new-look party.

Johnson is using the populist playbook, inspired by his pals Steve Bannon and Donald Trump and in readiness for going to “the people” if he fails to deliver on Brexit by Halloween. Truth is expendable, detail does not matter, reality is for defeatists, facts for losers. This shameless approach to politics sets out to harden divisions while relying on bluster, bravado and blaming other people for setbacks. The most significant lines in Johnson’s dismal speech on the steps of Downing Street were those talking about Brussels refusing to negotiate, clearly preparing the ground to shrug off personal responsibility for the disaster of no deal.

Sadly, the trajectory for the Tories could not be clearer. The liberals have been routed, moderates defeated and doubters despatched to the backbenches – even if they are Jeremy Hunt, runner-up in the leadership fight, or Penny Mordaunt, a popular Brexiteer. Now we witness a political entity famed for pragmatism, fiscal responsibility and ruthless pursuit of power turning into a repellent nationalist force.

The pugilistic Conservative party of Winston Churchill, Margaret Thatcher and Benjamin Disraeli is so scared by Nigel Farage’s return it is morphing into the Brexit party under its latest prime minister. This is why Priti Patel, a woman disgraced for undermining British diplomacy in the highly sensitive Middle East, and who until recently backed hanging, has been sent into the Home Office to toughen up their law and order credentials. This tactic might even win a snap election, especially if a splurge of cash is chucked around to win applause on police numbers, schools and social care – and Labour is foolish enough to keep Jeremy Corbyn as leader while failing to clarify its muddled position on Brexit.

But it is great news for Jo Swinson, the new Liberal Democrat leader, if she plays her cards right. Pandering to populism will drive away younger generations – which in Tory terms means under 51 – on whom their future depends, in order to lure a share of the electorate shrinking by 1% each year. And the lessons from countries such as France and Spain show how this sort of defeatist strategy sheds many moderates while inevitably failing to outflank forces even further to the right.

It is depressing to watch – made worse by the appeasers and captives who argued so strongly against no deal then instantly allowed personal ambition to override national interests. It seems the likes of Amber Rudd, Matt Hancock, Nicky Morgan and Jo Johnson have decided to ditch their principles for higher pay and power. Yet this government presents a test for every Conservative, just as Trump has tested all Republicans: do they endorse crass populism, regardless of cost to the country, or display the moral fibre to stand firm for their principles? History will judge turncoats harshly as any semblance of modernisation shrivels and dies.

• Ian Birrell is a former deputy editor of the Independent