When I started writing a column in the Guardian, I would engage with the commenters who made valid points and urge those whose response was getting lost in rage to re-read the piece and return. Comments were open for 72 hours. Coming up for air at the end of a thread felt like mooring a ship after a few days on choppy waters, like an achievement, something that I and the readers had gone through together. We had discussed sensitive, complicated ideas about politics, race, gender and sexuality and, at the end, via a rolling conversation, we had got somewhere.

In the decade since, the tenor of those comments became so personalised and abusive that the ship often drowned before making it to shore – the moderators would simply shut the thread down. When it first started happening, I took it as a personal failure – perhaps I had not struck the right tone or not sufficiently hedged all my points, provoking readers into thinking I was being dishonest or incendiary. In time, it dawned on me that my writing was the same. It was the commenters who had changed. It was becoming harder to discuss almost anything without a virtual snarl in response. And it was becoming harder to do so if one were not white or male.

As a result, the Guardian overhauled its policy and decided that it would not open comment threads on pieces that were certain to derail. The moderators had a duty of care to the writers, some of whom struggled with the abuse, and a duty of care to new writers who might succumb to a chilling effect if they knew that to embark on a journalism career nowadays comes inevitably with no protection from online thuggery. Alongside these moral concerns there were also practical, commercial ones. There were simply not enough resources to manage all the open threads at the same time with the increased level of attention that was now required.

In the past 10 years, many platforms in the press and social media have had to grapple with the challenges of managing users with increasingly sharp and offensive tones, while maintaining enough space for expression, feedback and interaction. Speech has never been more free or less intermediated. Anyone with internet access can create a profile and write, tweet, blog or comment, with little vetting and no hurdle of technological skill. But the targets of this growth in the means of expression have been primarily women, minorities and LGBTQ+ people.

A 2017 Pew Research Center survey revealed that a “wide cross-section” of Americans experience online abuse, but that the majority was directed towards minorities, with a quarter of black Americans saying they have been attacked online due to race or ethnicity. Ten per cent of Hispanics and 3% of whites reported the same. The picture is not much different in the UK. A 2017 Amnesty report analysed tweets sent to 177 female British MPs. The 20 of them who were from a black and ethnic minority background received almost half the total number of abusive tweets.

The vast majority of this abuse goes unpunished. And yet it is somehow conventional wisdom that free speech is under assault, that university campuses have succumbed to an epidemic of no-platforming, that social media mobs are ready to raise their pitchforks at the most innocent slip of the tongue or joke, and that Enlightenment values that protected the right to free expression and individual liberty are under threat. The cause of this, it is claimed, is a liberal totalitarianism that is attributable (somehow) simultaneously to intolerance and thin skin. The impulse is allegedly at once both fascist in its brutal inclinations to silence the individual, and protective of the weak, easily wounded and coddled.

This is the myth of the free speech crisis. It is an extension of the political-correctness myth, but is a recent mutation more specifically linked to efforts or impulses to normalise hate speech or shut down legitimate responses to it. The purpose of the myth is not to secure freedom of speech – that is, the right to express one’s opinions without censorship, restraint or legal penalty. The purpose is to secure the licence to speak with impunity; not freedom of expression, but rather freedom from the consequences of that expression.

The myth has two components: the first is that all speech should be free; the second is that freedom of speech means freedom from objection.

The first part of the myth is one of the more challenging to push back against, because instinctively it feels wrong to do so. It seems a worthy cause to demand more political correctness, politeness and good manners in language convention as a bulwark against society’s drift into marginalising groups with less capital, or to argue for a fuller definition of female emancipation. These are good things, even if you disagree with how they are to be achieved. But to ask that we have less freedom of speech – to be unbothered when people with views you disagree with are silenced or banned – smacks of illiberalism. It just doesn’t sit well. And it’s hard to argue for less freedom in a society in which you live, because surely limiting rights of expression will catch up with you at some point. Will it not be you one day, on the wrong side of free speech?

There is a kernel of something that makes all myths stick – something that speaks to a sense of justice, liberty, due process and openness and allows those myths to be cynically manipulated to appeal to the good and well-intentioned. But challenging the myth of a free speech crisis does not mean enabling the state to police and censor even further. Instead, it is arguing that there is no crisis. If anything, speech has never been more free and unregulated. The purpose of the free-speech-crisis myth is to guilt people into giving up their right of response to attacks, and to destigmatise racism and prejudice. It aims to blackmail good people into ceding space to bad ideas, even though they have a legitimate right to refuse. And it is a myth that demands, in turn, its own silencing and undermining of individual freedom. To accept the free-speech-crisis myth is to give up your own right to turn off the comments.

At the same time that new platforms were proliferating on the internet, a rightwing counter-push was also taking place online. It claimed that all speech must be allowed without consequence or moderation, and that liberals were assaulting the premise of free speech. I began to notice it around the late 2000s, alongside the fashionable atheism that sprang up after the publication of Richard Dawkins’s The God Delusion. These new atheists were the first users I spotted using argumentative technicalities (eg “Islam is not a race”) to hide rank prejudice and Islamophobia. If the Guardian published a column of mine but did not open the comment thread, readers would find me on social media and cry censorship, then unleash their invective there instead.

As platforms multiplied, there were more and more ways for me to receive feedback from readers – I could be sworn at and told to go back to where I came from via at least three mediums. Or I could just read about how I should go back to where I came from in the pages of print publications, or on any number of websites. The comment thread seemed redundant. The whole internet was now a comment thread. As a result, mainstream media establishments began to struggle with this glut of opinion, failing to curate the public discussion by giving into false equivalence. Now every opinion must have a counter-opinion.

The dark side of Guardian comments Read more

I began to see it in my own media engagements. I would be called upon by more neutral outlets, such as the BBC, to discuss increasingly more absurd arguments with other journalists or political activists with extreme views. Conversations around race, immigration, Islam and climate change became increasingly binary and polarised even when there were no binaries to be contemplated. Climate change deniers were allowed to broadcast falsehoods about a reversal in climate change. Racial minorities were called upon to counter thinly veiled racist or xenophobic views. I found myself, along with other journalists, regularly ambushed. I appeared on BBC’s Newsnight to discuss an incident in which a far-right racist had mounted a mosque pavement with his car and killed one of the congregation, and I tried to make the point that there was insufficient focus on a growing far-right terror threat. The presenter then asked me: “Have you had abuse? Give us an example.” This became a frequent line of inquiry – the personalisation and provocation of personal debate – when what was needed was analysis.

It became common for me and like-minded colleagues to ask – when invited on to TV or radio to discuss topics such as immigration or Islamophobia – who was appearing on the other side. One British Asian writer was invited on to the BBC to discuss populist rage. When he learned that he would be debating Melanie Phillips – a woman who has described immigrants as “convulsing Europe” and “refusing to assimilate” – he refused to take part, because he did not believe the topic warranted such a polarised set-up. The editor said: “This will be good for your book. Surely you want to sell more copies?” The writer replied that if he never sold another book in his life as a result of refusing to debate with Melanie Phillips, he could live with that. This was now the discourse: presenting bigotry and then the defence of bigotry as a “debate” from which everyone can benefit, like a boxing match where even the loser is paid, along with the promoters, coaches and everyone else behind arranging the fight. The writer Reni Eddo-Lodge has called it “performing rage”.

Views previously consigned to the political fringes made their way into the mainstream via social and traditional media organisations that previously would never have contemplated their airing. The expansion of media outlets meant that it was not only marginalised voices that secured access to the public, but also those with more extreme views.

This inevitably expanded what was considered acceptable speech. The Overton window – the range of ideas deemed to be acceptable by the public – shifted as more views made their way from the peripheries to the centre of the conversation. Any objection to the airing of those views would be considered an attempt to curtail freedom of speech. Whenever I attempted to push back in my writing against what amounted to incitement against racial or religious minorities, my opponents fixated on the free speech argument, rather than the harmful ramifications of hate speech.

In early 2018, four extreme-right figures were turned away at the UK border. Their presence was deemed “not conducive to the public good”. When I wrote in defence of the Home Office’s position, my email and social media were flooded with abuse for days. Rightwing media blogs and some mainstream publications published pieces saying my position was an illiberal misunderstanding of free speech. No one discussed the people who were banned, their neo-Nazi views, or the risk of hate speech or even violence had they been let in.

What has increased is not intolerance of speech; there is simply more speech. And because that new influx was from the extremes, there is also more objectionable speech – and in turn more objection to it. This is what free-speech-crisis myth believers are picking up – a pushback against the increase in intolerance or bigotry. But they are misreading it as a change in free speech attitudes. This increase in objectionable speech came with a sense of entitlement – a demand that it be heard and not challenged, and the freedom of speech figleaf became a convenient tool. Not only do free speech warriors demand all opinions be heard on all platforms they choose, from college campuses to Twitter, but they also demand that there be no objection or reaction. It became farcical and extremely psychologically taxing for anyone who could see the dangers of hate speech, and how a sharpening tone on immigration could be used to make the lives of immigrants and minorities harder.

When Boris Johnson compared women who wear the burqa to “letterboxes” and “bank robbers”, it led to a spike in racist incidents against women who wear the niqab, according to the organisation Tell Mama, a national project which records and measures anti-Muslim incidents in the UK. Pointing this out and making the link between mockery of minorities and racist provocation against them was, according to Johnson’s supporters, assailing his freedom of speech. The British journalist Isabel Oakeshott tweeted that if he were disciplined by his party for “perfectly reasonable exercise of free speech, something has gone terribly wrong with the party leadership”, and that it was “deplorable to see [the Tory leadership] pandering to the whinings of the professionally offended in this craven way”.

Free speech had seemingly come to mean that no one had any right to object to what anyone ever said – which not only meant that no one should object to Johnson’s comments but, in turn, that no one should object to their objection. Free speech logic, rather than the pursuit of a lofty Enlightenment value, had become a race to the bottom, where the alternative to being “professionally offended” is never to be offended at all. This logic today demands silence from those who are defending themselves from abuse or hate speech. It is, according to the director of the Institute of Race Relations, “the privileging of freedom of speech over freedom to life”.

Our alleged free speech crisis was never really about free speech. The backdrop to the myth is rising anti-immigration sentiment and Islamophobia. Free-speech-crisis advocates always seem to have an agenda. They overwhelmingly wanted to exercise their freedom of speech in order to agitate against minorities, women, immigrants and Muslims.

But they dress these base impulses up in the language of concern or anti-establishment conspiracism. Similar to the triggers of political-correctness hysteria, there is a direct correlation between the rise in free speech panic and the rise in far-right or hard-right political energy, as evidenced by anti-immigration rightwing electoral successes in the US, the UK and across continental Europe. As the space for these views expanded, so the concept of free speech became frayed and tattered. It began to become muddled by false equivalence, caught between fact and opinion, between action and reaction. The discourse became mired in a misunderstanding of free speech as absolute.

Facebook Twitter Pinterest Donald Trump signs an executive order requiring US colleges and universities to ‘support free speech’. Photograph: UPI/Barcroft

As a value in its purest form, freedom of speech serves two purposes: protection from state persecution, when challenging the authority of power or orthodoxy; and the protection of fellow citizens from the damaging consequences of absolute speech (ie completely legally unregulated speech) such as slander. According to Francis Canavan in Freedom of Expression: Purpose As Limit – his analysis of perhaps the most permissive free speech law of all, the first amendment of the US constitution – free speech must have a rational end, which is to facilitate communication between citizens. Where it does not serve that end, it is limited. Like all freedoms, it ends when it infringes upon the freedoms of others. He writes that the US supreme court itself “has never accepted an absolutist interpretation of freedom of speech. It has not protected, for example, libel, slander, perjury, false advertising, obscenity and profanity, solicitation of a crime, or ‘fighting’ words. The reason for their exclusion from first-amendment protection is that they have minimal or no values as ideas, communication of information, appeal to reason, step towards truth etc; in short, no value in regard to the ends of the amendment.”

Those who believe in the free-speech-crisis myth fail to make the distinction between “fighting” words and speech that facilitates communication; between free speech and absolute speech. Using this litmus test, the first hint that the free speech crisis is actually an absolute speech crisis is the issues it focuses on. On university campuses, it is overwhelmingly race and gender. On social media, the free speech axe is wielded by trolls, Islamophobes and misogynists, leading to an abuse epidemic that platforms have failed to curb.

This free speech crisis movement has managed to stigmatise reasonable protest, which has existed for years without being branded as “silencing”. This is, in itself, an assault on free expression.

What is considered speech worthy of protection is broadly subjective and depends on the consensual limits a society has drawn. Western societies like to think of their version of freedom of speech as exceptionally pristine, but it is also tainted (or tempered, depending on where you’re coming from) by convention.

There is only one way to register objection of abhorrent views, which is to take them on. This is a common narcissism in the media. Free speech proponents lean into the storm, take on the bad guys and vanquish them with logic. They also seem, for the most part, incapable of following these rules themselves.

Bret Stephens of the New York Times – a Pulitzer prize-winning star columnist who was poached from the Wall Street Journal in 2017 – often flatters himself in this light, while falling apart at most of the criticism he receives. For a man who calls for “free speech and the necessity of discomfort” as one of his flagship positions as a columnist, he seems chronically unable to apply that discipline to himself.

In his latest tantrum, just last week, Stephens took umbrage against a stranger, the academic David Karpf, who made a joke calling him a “metaphorical bedbug” on Twitter, as a riff on a report that the New York Times building was suffering from a bedbug infestation. (The implication was that Stephens is a pain and difficult to get rid of, just to kill the punchline completely.)

Stephens was alerted to the tweet, then wrote to Karpf, his provost, and the director of the School of Media and Public Affairs, where Karpf is a professor. He in effect asked to speak to Karpf’s managers so that he could report on a man he doesn’t know, who made a mild joke about him that would otherwise have been lost in the ether of the internet because – well, because, how dare he? The powerful don’t have to suffer “the necessity of discomfort”; it’s only those further down the food chain who must bear the moral burden of tolerance of abusive speech. Stephens’s opponents – who include Arabs, whose minds Stephens called “diseased”, and Palestinians, who are en masse one single “mosquito” frozen in amber – must bear it all with good grace.

Stephens has a long record of demanding respect when he refuses to treat others with the same. In response to an objection that the New York Times had published an article about a Nazi that seemed too sympathetic, he wrote: “A newspaper, after all, isn’t supposed to be a form of mental comfort food. We are not an advocacy group, a support network, a cheering section, or a church affirming a particular faith – except, that is, a faith in hard and relentless questioning.” He called disagreement “a dying art”. This was particularly rich from someone who at one time left social media because it was too shouty, only to return sporadically to hurl insults at his critics.

In June 2017, Stephens publicly forswore Twitter, saying that the medium debased politics and that he would “intercede only to say nice things about the writing I admire, the people I like and the music I love”.

He popped up again to call ex-Obama aide Tommy Vietor an “asshole” (a tweet he later deleted after it was flagged as inappropriate by the New York Times). In response to a tweet by a Times colleague (who had himself deleted a comment after receiving flack for it, and admitted that it had not been well crafted), Stephens said: “This. Is. Insane. And must stop. And there is nothing wrong with your original tweet, @EricLiptonNYT. And there is something deeply psychologically wrong with people who think there is. And fascistic. And yes I’m still on Twitter.”

A dying art indeed. Stephens again deactivated his account after bedbug-gate, retreating to the safe space of the high security towers of the New York Times where, I am told, the bedbug infestation remains unvanquished.

Stephens is a promoter of the “free speech crisis” myth. It is one that journalists, academics and political writers have found useful in chilling dissent. The free-speech-crisis myth serves many purposes. Often it is erected as a moral shield for risible ideas – a shield that some members of the media are bamboozled into raising because of their inability to look past their commitment to free speech in the abstract.

Trolling has become an industry. It is now a sort of lucrative contact sport, where insults and lies are hurled around on television, radio, online and in the printed press. CNN’s coverage of the “Trump transition”, after Donald Trump was elected as US president, was a modern version of a medieval freak show. Step right up and gawk at Richard Spencer, the Trump supporter and head of far-right thinktank the National Policy Institute, as he questions whether Jews “are people at all, or instead soulless golem”. And at the black Trump surrogate who thinks Hillary Clinton started the war in Syria. And at Corey Lewandowski, a man who appeared on CNN as a political commentator, who appears to make a living from lying in the media, and who alleged that the Trump birther story, in which Trump claimed that Barack Obama was not born on US soil, was in fact started by Hillary Clinton.

In pursuit of ratings – from behind a “freedom of speech” figleaf, and perhaps with the good intention of balance on the part of some – many media platforms have detoxified the kind of extreme or untruthful talk that was until recently confined to the darker corners of Reddit or Breitbart. And that radical and untruthful behaviour has a direct impact on how safe the world is for those smeared by these performances. Trump himself is the main act in this lucrative show. Initially seen as an entertaining side act during his election campaign, his offensive, untruthful and pugnacious online presence became instantly more threatening and dangerous once he was elected. Inevitably, his incontinence, bitterness, rage and hatemongering, by sheer dint of constant exposure, became less and less shocking, and in turn less and less beyond the pale.

A world where all opinions and lies are presented to the public as a sort of take-it-or-leave it buffet is often described as “the marketplace of ideas”, a rationalisation for freedom of expression based on comparing ideas to products in a free-market economy. The marketplace of ideas model of free speech holds that what is true factually, and what is good morally, will emerge after a competition of ideas in a free, unmoderated and transparent public discourse, a healthy debate in which the truth will prevail. Bad ideas and ideologies will lose out and wither away as they are vanquished by superior ones. The problem with the marketplace of ideas theory (as with all “invisible hand”-type theories) is that it does not account for a world in which the market is skewed, and where not all ideas receive equal representation because the market has monopolies and cartels.

But real marketplaces actually require a lot of regulation. There are anti-monopoly rules, there are interest rate fixes and, in many markets, artificial currency pegs. In the press, publishing and the business of ideas dispersal in general, there are players that are deeply entrenched and networked, and so the supply of ideas reflects their power.

Freedom of speech is not a neutral, fixed concept, uncoloured by societal prejudice. The belief that it is some absolute, untainted hallmark of civilisation is linked to self-serving exceptionalism – a delusion that there is a basic template around which there is a consensus uninformed by biases. The recent history of fighting for freedom of speech has gone from something noble – striving for the right to publish works that offend people’s sexual or religious prudery, and speaking up against the values leveraged by the powerful to maintain control – to attacking the weak and persecuted. The effort has evolved from challenging upwards to punching downwards.

It has become bogged down in false equivalence and extending the sanctity of fact to opinion, thanks in part to a media that has an interest in creating from the discourse as much heat as possible – but not necessarily any light. Central in this process is an establishment of curators, publishers and editors for whom controversy is a product to be pushed. That is the marketplace of ideas now, not a free and organic exchange of intellectual goods.

The truth is that free speech, even to some of its most passionate founding philosophers, always comes with braking mechanisms, and they usually reflect cultural bias. John Milton advocated the destruction of blasphemous or libellous works: “Those which otherwise come forth, if they be found mischievous and libellous, the fire and the executioner will be the timeliest and the most effectual remedy, that mans [sic] prevention can use.” Today, our braking mechanisms still do not include curbing the promotion of hate towards those at the bottom end of the social hierarchy, because their protection is not a valued or integral part of our popular culture – despite what the free-speech-crisis myth-peddlers say.

Free speech as an abstract value is now directly at odds with the sanctity of life. It’s not merely a matter of “offence”. Judith Butler, a cultural theorist and Berkeley professor, speaking at a 2017 forum sponsored by the Berkeley Academic Senate, said: “If free speech does take precedence over every other constitutional principle and every other community principle, then perhaps we should no longer claim to be weighing or balancing competing principles or values. We should perhaps frankly admit that we have agreed in advance to have our community sundered, racial and sexual minorities demeaned, the dignity of trans people denied, that we are, in effect, willing to be wrecked by this principle of free speech.”

Political correctness: how the right invented a phantom enemy | Moira Weigel Read more

We challenge this instrumentalisation by reclaiming the true meaning of the freedom of speech (which is freedom to speak rather than a right to speak without consequence), challenging hate speech more forcefully, being unafraid to contemplate banning or no-platforming those we think are harmful to the public good, and being tolerant of objection to them when they do speak. Like the political-correctness myth, the free-speech-crisis myth is a call for orthodoxy, for passiveness in the face of assault.

A moral right to express unpopular opinions is not a moral right to express those opinions in a way that silences the voices of others, or puts them in danger of violence. There are those who abuse free speech, who wish others harm, and who roll back efforts to ensure that all citizens are treated with respect. These are facts – and free-speech-crisis mythology is preventing us from confronting them.

This is an edited extract from We Need New Stories: Challenging the Toxic Myths Behind Our Age of Discontent, published by W&N on 5 September and available at guardianbookshop.co.uk

• Follow the Long Read on Twitter at @gdnlongread, and sign up to the long read weekly email here.