A coalition of free speech organisations rallied together last week to defend Simon & Schuster’s choice to publish professional irritant Milo Yiannopoulos’s autobiography, Dangerous, saying that boycotting the book, as so many people have called for, would have “a chilling effect” on free speech.

I’m sure that having his book (which hit the No 1 spot on Amazon’s pre-sale charts the day after it was announced) pulled from shelves or dropped from S&S would catapult it to even greater success at another publisher, but never mind. It’s clear that this coalition of organisations are standing up for what they believe in, and feel it is important to defend Yiannopoulos’ well-rehearsed right to speak his mind.

Defending free speech often means finding yourself in the difficult position of having to defend people who say disgusting things. People who make jokes about rape; Holocaust deniers; straight-up racists. Though we might not like what these people say, it’s important that they’re allowed to say it. You can’t go around censoring people just because you don’t agree with them. If we can’t all express what we think, then we can’t talk to each other about our ideas. We can’t have a discussion; we can’t improve; we can’t function as a society.

And yet. Like the tax lawyer I met at a party who insisted several times that there was “nothing illegal at all” about what she did for a living, I can’t help but feel that Yiannopoulos and his ilk are protesting too much when they say something Islamophobic or misogynistic and insist that they are protected by “free speech”. And I’m uncertain that these organisations standing up to protect Yiannopoulous are doing the right thing – especially when what he says, and the people he says it for, are doing so much to poison reasoned discussion.

For a start, Simon & Schuster being criticised for publishing the book is hardly censorship. Anyone can write and publish a book without the help of a major publishing house these days – and can be very successful at it too. The man works for a news site, makes numerous media appearances, writes a regular column, and frankly if you think that the world lacks for his opinions in any way at the moment then please, please tell me which parallel universe you are living in so I can move there.

Free speech has limits. You aren’t allowed to shout “fire” in a crowded theatre because someone’s probably going to get hurt. Your right to say what you like is trumped by your responsibility to stop me being trampled to death by a stampede of panicked theatre-goers. Death threats; rape threats; bomb threats; online abuse that drives someone to suicide – these are all things that free speech doesn’t cover – and which aren’t appropriate to defend in its name. Doing so makes it even harder for people to speak freely – not least because that idea of “speaking freely” becomes co-opted by people who mistake it for “I should be able to shout ‘free speech’ at you until you stop talking”.

Self-proclaimed super-villain Yiannopoulos has made a living from saying and doing hateful things, and has successfully embroiled himself in numerous headline-grabbing controversies. Whether it’s saying that “gay rights have made us dumber”, calling transgender people “mentally ill”, calling rape culture “a fantasy”, or being banned by Twitter for allegedly encouraging trolls to attack Ghostbusters actor Leslie Jones with a tirade of racist and sexist abuse, you can usually find him saying something pathologically awful.

So when a major publishing house pays $250,000 to print the work of an alt-right figurehead like him, it gives credence to these ideas, and makes them part of the mainstream. It endorses them. It empowers everyone who agrees with them to act on their worst impulses, and spread hate speech.

Hate speech is not compatible with reasoned debate. You can’t talk to it. When you try, it talks over you and ignores you and calls you a fat ugly whore and publishes your address online. If you’re not scared out of engaging with it for fear of reprisal, chances are you’ll die of exhaustion. How many times do you have to explain to people that “racism is bad” or “women are not worse than men” before you give up because it’s not worth the bother? These are not discussions worth having. They shouldn’t even be discussions.

With the ascendancy of Donald Trump and the rise of the alt-right we have seen a dangerous normalisation of ideas that we would once have rejected as too outrageous to endorse with our attention. We now publish endless think-pieces on them, have panel discussions about them, and publishing houses pay a quarter of a million dollars for someone to write a book about them.

For free speech organisations to stand up and defend Yiannopoulos is a dangerous miscalculation. It further supports the narrative that he’s a radical visionary being oppressed by the system, rather than an opinion-spamming hack who brings out the worst in people. In their statement, the coalition write that “only vigorous disagreement can counter toxic speech effectively”. But if you’ve been on the internet lately, you’ll know that we’re not short of vigorous disagreement, and that there’s more toxic speech than ever.

Free speech is vital. Of course it is. But is defending it like this really helping?