When historians write the last pages of their books, and the producers of history documentaries sit down to edit the final minutes of their programmes, there is often a strong urge to look to the future and emphasise the positive. That impulse is at its most acute when the subject of the book or documentary in question is British history.

The classic formulation is something along the lines of: Britain and her institutions are remarkably adaptable, the nation has survived two world wars and endured the loss of empire, we will surely cope with whatever the future has in store… roll credits, press send, everyone down the pub.

The past three years have left many of those final chapters and closing scenes in need of urgent rewrites. Brexit, the rise of populism and the constitutional crisis in which we are still utterly ensnared – despite the passing distraction of the Tory leadership circus – undermines that sort of blithe optimism.

Now is not the time for upbeat endings. It is a moment to make the case for an ever unpopular and always controversial sentiment – pessimism.

But pessimism has an image problem. To describe someone as a pessimist is to issue an insult, whereas to be labelled an optimist is to get a pat on the back. To dismiss someone’s argument as pessimistic is to suggest it is the product of a personality disorder, rather than careful analysis.

Key moments from Tory candidates' interviews with Andrew Neil Read more

This lazy stance is easy to adopt and often highly effective in debate. It is what Boris Johnson clumsily and desperately attempted last week, while floundering against Andrew Neil: “Why this defeatism? Why this negativity?” he blustered, in a pitiful effort to draw attention away from his demonstrable ignorance of his own Brexit “plan”. Don’t analyse, stop identifying flaws and inconsistencies, just be optimistic. Rejoice. Rejoice.

The prevailing cult of optimism reinforces the belief that Britain’s institutions – parliament, the civil service and that jumble of conventions and archaic procedures that are what passes for a constitution – will inevitably weather any future storm.

Well, it’s not been a great week for the civil service; a police investigation has been launched into a leak apparently designed to bring down our ambassador to the most powerful nation on earth, followed by his public defenestration by the PM-in-waiting. Month by month our constitution has been proved unfit for purpose and parliament’s physical decay is increasingly turning it into a vast, scaffold-covered metaphor.

A musty, chintzy kitschness lingers about the Palace of Westminster. A cabbagey, care-home smell wafts along its neo-gothic corridors. With Johnson refusing to rule out bypassing parliament (with his threatened prorogation and his opponents discussing setting up a rival assembly across the road, you have to really want to see the “mother of parliaments” cup as half full.

The unlikely patron saint of this form of optimistic, national exceptionalism is Winston Churchill, a man who suffered bouts of depression and spent a decade in the political wilderness for pessimistically predicting a global catastrophe. Hardly the ideal poster boy for the breezy optimism of Johnson and his cabal. And to go by run of form, optimism is on a losing streak. Optimists confidently predicted Trump would mature in office and they have repeatedly administered the last rites to Nigel Farage’s political career and failed utterly to identity the risks that led us into our current age of anger and chaos.

Whether we like it or not, there are moments in history when pessimism is the appropriate response. Times when, as the German philosopher Oswald Spengler said, “optimism is cowardice”. What is needed now is not a “nameless, unreasoning, unjustified” form of pessimism that “paralyses needed efforts to convert retreat into advance”, to quote Franklin D Roosevelt, but a sobering and energising pessimism. It is necessary because the cult of optimism, the original source of our national complacency, is in itself a clear and present danger.

Optimists predicted Trump would mature in office and have repeatedly administered the last rights to Nigel Farage’s political career

The Brexit project exploited our cultural predilection for optimism. Leave was painted as the optimistic choice. Now, when the Brexiteers are not promising us “adequate food”, they are peddling another brand of optimism. No matter what happens at the end of October, they tell us, we’ll be all right. After all, if we can make it through the Second World War we can survive Brexit. One of the many holes in this “there’ll always be an England” line of argument is, of course, that half a million British people didn’t make it through the Second World War.

That reality is perhaps why the support for Remain was high among a demographic that is rarely mentioned. The simplistic belief that the old voted Leave and the young voted Remain ignores the fact that the most elderly, the people who actually remember the Second World War, who fought and suffered in it, were “far more likely to oppose Brexit”, according to some research, than the baby boomers – a generation brought up watching war films rather than cowering in Anderson shelters.

And is it that surprising that the most privileged generation to have ever lived harbour less pessimism than their parents or their children and grandchildren?

It is true that Britain and its institutions have survived past crises, but often this was because those in charge, at a certain point, snapped out of the stupor of latent optimism, recognised the dangers circling the nation and acted. Call it pessimism, realism or pragmatism: we urgently need exactly that sort of awakening now.

• David Olusoga is a historian and broadcaster