I felt a new emotion yesterday: empathy with the Conservative MP and arch-Remainer Anna Soubry. In response to protesters on College Green who were chanting ‘Soubry is a Nazi!’ as she was being interviewed by the BBC, Soubry said: ‘I do object to being called a Nazi.’ And I knew just how she felt. Many Leave voters will have. Because for two-and-a-half years we have been similarly libelled as Nazis, branded by everyone from Prince Charles to the Archbishop of Canterbury to broadsheet columnists as racist, xenophobic cretins whose self-destructive vote to leave the EU has resuscitated the ‘fascist tradition’ (in the archbishop’s words).

So we know how you feel, Ms Soubry. And we know it isn’t nice. To be denounced as fascistic simply because you hold a political opinion that others do not like is horrible and demeaning. I am glad it has only happened to you a couple of times, whereas for us Leavers it has felt like a daily occurrence. There it is, day in, day out, in the mutterings of Remain-leaning politicians, in the bilious output of Lord Adonis (leaving the EU is comparable to ‘appeasement in the 1930s’, he said), in the increasingly frenzied claims made by sections of the academy (Brexit is the ‘harbinger of a return to the 1930s’, professors insist) – time and again that ugly slur that says the 17.4million who voted for Brexit either have Nazi leanings or certainly helped to bring about a Nazi-like climate. But here’s a question, a question that demands an answer from all of those who are up in arms about the horrible ‘Nazi’ chants made at Ms Soubry yesterday. People like Norman Smith of the BBC, for example, who asked: ‘Is this what it’s come to… Nazi taunts?’ The question is this: why are you only now offended by Nazi taunts? Where was your outrage during the ceaseless, background noise of Nazi taunts against Leave voters over the past two-and-a-half years? Why do gruff blokes shouting ‘Nazi’ at Anna Soubry offend you more than bishops and princes of the realm and well-educated columnists effectively doing the same to ordinary voters, though of course in expertly crafted speeches and pristinely worded newspaper columns rather than in rough bellowing from a public green?

Indeed, not only have the chattering classes suddenly discovered that it is unpleasant to refer to people who aren’t Nazis as Nazis – they have even demanded that the police take action against such uncalled-for Nazi taunts. According to reports, the Metropolitan Police have launched an investigation not only into protesters’ surrounding of Ms Soubry as she walked into the Commons but also of their earlier cry of ‘Soubry is a Nazi’. In the words of the Press Association, ‘Police are investigating whether any criminal offence was committed when Conservative MP Anna Soubry was branded a “Nazi” by Brexit protesters outside parliament’. There are a few things to say about this. First, what kind of country criminalises the insulting of politicians? An unfree one. Indeed, it would be profoundly ironic if the authorities responded to the branding of a politician as fascistic by arresting and charging people for the crime of expressing their political opinions. Secondly, if it is now a crime to shout ‘Nazi’ or ‘fascist’ at people who clearly are neither, then the entire Corbynista left had better engage the services of a good lawyer: overblown fascist accusations are their stock-in-trade.

And thirdly, and most pressingly, there’s that question that hangs darkly over all of this, which none of the political or media class can answer, at least not without exposing their own prejudices: why is it a crime to issue Nazi taunts against Anna Soubry but not against the housewife in Stoke or the former miner in Wales or the beauty therapist in Essex who all voted for Brexit? ‘Fascists’, the chattering classes have been whispering and hinting about these sorts of people since June 2016, and the police showed not the slightest bit of interest in those Nazis taunts. As well they should not. Speech should never be a police matter. Including heated speech, angry speech, protesting speech. Including both speech that libels Leave voters and speech that mauls Remainers. Including even the speech that was heard in Westminster yesterday, however unpleasant it might have been. The response to the protests against Anna Soubry, and also against the Guardian columnist Owen Jones, has been far worse than the protests themselves. First because of the way the infinitesimally small band of irate protesters are being held up as archetypal Brexiteers, as if these largely hard-right individuals represent the 17.4million who voted for Brexit – the largest electoral bloc in UK history. They do not. Brexit was an accomplishment of vast numbers of decent, hard-working, good people, and to demean it as the handiwork of cranks and blowhards is an insult to the demos.

And the second reason we should worry about the response to the protest is because it has been alarmingly authoritarian. Lord Adonis has written to the police to find out why they didn’t take action against the protesters. MPs want the police to clamp down on Brexit protesting outside the Commons. Even supposedly radical Corbynistas suggest tougher action is needed against ‘the fash’, as they see pretty much everyone who disagrees with them. These calls are incredibly dangerous. Parliament Square and College Green and other areas around the Houses of Parliament have been sites of protest for centuries – from the December Days of 1641, when apprentices from across London gathered in rowdy fashion to demand the expulsion of bishops from the House of Lords, to Black Friday in November 1910, when Suffragettes gathered outside parliament and were denounced by that era’s guardians of moral opinion as a ‘mob’ (sound familiar?) and were savagely beaten by the police for having had the temerity to agitate so near to parliament. To restrict the right to protest around parliament just because disagreeable people are currently protesting there would be devastating to freedom of association and freedom of speech.

So it is unfortunate that Anna Soubry was subjected to verbal abuse. Just as it is unfortunate that Leavers have been subjected to similar abuse, though on an even larger, more constant scale. But both us Leavers and Remainers like Ms Soubry must accept the following fact, difficult as it might sometimes be: people’s right to express their political anger must take precedence over our sensitivities. Every single time. Brendan O’Neill is editor of spiked and host of the spiked podcast, The Brendan O’Neill Show. Subscribe to the podcast here. And find Brendan on Instagram: @burntoakboy