In a packed tent at the Festival of Ideas at Kenwood, a member of the alarmed audience recently put to me a chilling question: how far does the language of political violence result in actual violence?

Tellingly, he asked me this even before the Supreme Court judgement on Boris Johnson’s unlawful prorogation of parliament, before the prime minister started outrageously bringing the memory of Jo Cox into a twisted defence of his own positions, and before he started dismissing the abuse and death threats directed at some women MPs.

The bitter exchanges in parliament in recent days have sustained the potential eruption of violence over Brexit as a live issue. There have been – mostly anonymous – warnings from Brexiteers that there will be “serious civil unrest” if they are not permitted to “get Brexit done”.

What’s truly shocking is that the government and its supporters in the media are using the very real aggression of a small number of people, directed against the many opposed to Brexit, to foreshadow an imaginary, all-out civil war if passports are not blue by 1 November.

At the festival, I therefore tried to be reassuring. Despite the deeply unpleasant nature of some public discourse – with conscientious public servants vilified as traitors and saboteurs, and government threatening to ignore acts of parliament – the UK remains a long way from a public breakdown of law and order. We are nowhere near even the sporadic violent protests by the gilets jaunes (yellow vests) in France, which are now largely spent.

Police separate clashing Brexit protesters outside Parliament Show all 11 1 /11 Police separate clashing Brexit protesters outside Parliament Police separate clashing Brexit protesters outside Parliament Anti-Brexit protester Steve Bray (left) and a pro-Brexit protester argue as they demonstrate outside the Houses of Parliament Getty Images Police separate clashing Brexit protesters outside Parliament A pro-Brexit protester argues Getty Images Police separate clashing Brexit protesters outside Parliament Police surround the pro-Brexit protester after he confronted Steve Bray, a pro-European protester Getty Images Police separate clashing Brexit protesters outside Parliament A leave supporter is spoken to by a police officer as he argues with a remain supporter, Steve Bray, outside Parliament PA Police separate clashing Brexit protesters outside Parliament MPs in Parliament are to vote on Theresa May's Brexit deal next week after her December vote was called off in the face of a major defeat Getty Images Police separate clashing Brexit protesters outside Parliament A leave supporter is spoken to by a police officer PA Police separate clashing Brexit protesters outside Parliament Police look on as anti-Brexit protesters demonstrate outside the Houses of Parliament. Getty Images Police separate clashing Brexit protesters outside Parliament Police hold back a leave supporter PA Police separate clashing Brexit protesters outside Parliament A police officer speaks with anti-Brexit protester Steve Bray Getty Images Police separate clashing Brexit protesters outside Parliament Police surround a pro-Brexit protester after he confronted a pro-European protester Getty Images Police separate clashing Brexit protesters outside Parliament Conservative MP David Davies, wearing a gopro camera, speaks to anti-Brexit protester Steve Bray Getty Images

What we must remember is that there’s a difference between private and public anger.

My father was an angry, rather right-wing Conservative. In the 1950s, he was driven to fury by what he saw as treacherous anti-British agitators like Egypt’s Colonel Nasser, who “stole” “our” Suez Canal, or “communist terrorists” like Jomo Kenyatta, who determined to transform Kenya from a British colony to an independent republic. His anger took the form of shouting at my mother and me, and occasionally of throwing missiles at the radio.

Today, the line between that private anger and a public outburst is less distinct than it was. Those shouting at the television can – and do – take to social media. Some with an axe to grind hide behind anonymous avatars to troll their chosen hate figures.

This is very nasty, and it can foreshadow worse. But we cannot allow the fear of horrendous but rare incidents to make us behave as if mass political violence is commonplace.

Despite being the leader of the main explicitly pro-Remain party for two years, I encountered none of the physical aggression – such as spitting in the streets – which my colleagues and I experienced in the early years of the coalition.

Many MPs have long had to put up with abuse – in my case, the episodes of vandalism, hate mail, even occasional excrement through the door that have punctuated my political career even in a generally very sedate part of suburban London.

Many women MPs have had a much rougher time as the targets of cowardly men who express their inadequacy through very nasty misogyny. The same goes for BAME MPs and vile racism. What they have been on the receiving end of is abhorrent, and the perpetrators must be pursued and prosecuted. But the Brexiteers cannot get away with telling us that these incidents mean widespread violence on behalf of 17.4 million (mostly quite old) Leave voters is just around the corner.

A much more credible and serious threat to the public than Brexiteer outrage are the violent extremists incubated by fringe political groups. Until recently Islamic militants were the main threat. Now, according to the security services, right-wing racist groups are becoming a more imminent danger. The danger now is that people on this dark fringe might talk themselves into violence – and the example of Jo Cox shows us where this kind of violence can lead.

Brexit has undoubtedly fuelled extreme nationalism and xenophobia, and there has been a rise in reported hate crime. Yet we must keep in perspective the fact that many black and Asian Britons experienced much worse in the wake of Powell’s “Rivers of Blood” speech and subsequent eruptions of racism. More recently, there has been a general uptick in Islamophobia.

But memories are short. The UK has seen much more serious civil unrest than anything that is apparent today: the miners’ strike and Orgreave; the Poll Tax riots; Toxteth and Brixton; and, within the last decade, the 2011 riots in Croydon, Tottenham and elsewhere. And that’s ignoring the deadly political violence of the Troubles, which claimed well over 3000 lives.

This amnesia is, of course, exactly what Boris Johnson and Dominic Cummings want to exploit. The threat of violence is a cynical excuse to short-circuit the processes of law and parliament by insisting that the government needs to defuse a “national emergency”. They aren’t the first to try this: I recall Theresa May warning that there would be riots if we were to have a confirmatory referendum. If so, they would be the first protests in history against the outrage of being given a vote.

Support free-thinking journalism and attend Independent events

Precedent suggests that some measure of unrest would be more likely if the prime minister’s brinkmanship badly disrupts our food, medical and fuel supplies. Two million young people who have joined the electoral register since 2016, and the several million people – mainly students – who were missed off it in 2016 would have particular cause for protest.

Whatever Johnson says, Brexit will not be resolved by banging the table, or by frightening people with the idea that 17.4 million righteous, frustrated, Brexiters will take the law into their own hands in a very unevenly matched national punch-up with the 29 million people who didn’t vote to leave the EU. They won’t.

In the absence of a parliamentary majority either for the original withdrawal agreement, or for the prime minister’s latest “border zone” deal – less than a week old, and already all but dismissed – there remains only one sensible way to end the impasse: a confirmatory referendum on whichever deal the EU will accept, with the alternative option being to Remain.

Whatever the outcome, I have no doubt that the public would insist on a peaceful and respectful debate and put the hate-peddlers to shame.