The Home Office doesn’t often get it right – but by declining to indulge the muddled preciousness that surrounds the freedom of speech debate, it has done so. In the space of a fortnight, four extreme-right figures have been turned away at the UK border. The latest was the founder of the German far-right group Pegida. On Monday Lutz Bachmann was denied entry to the UK, and deported. He had been due to give a speech at Speakers’ Corner in London’s Hyde Park in London, but his presence was deemed “not conducive to the public good”.

Bachmann was to have addressed a “free speech” rally. It’s not clear how this was different to any other rally, other than by framing any opposition to it as censorship. Characters such as Bachmann are no innocents practising their freedom of speech: they are cynical exploiters of it. They’re little better than loiterers waiting round the corner to jump on your windshield, pretending to be hurt, shaking you down for money. It’s a scam, trading notoriety and worse for attention. Why do we fall for it?

Most freedom of speech debates now start on the false premise that denying someone a platform is censorship. So we must begin with the correct one, which is that freedom of speech is freedom from punishment. If you are not being convicted and penalised by the state for speaking, then you have freedom of speech. If just one channel of speech has been denied to you, you still have freedom of speech. We’re not talking pulping Lady Chatterley’s Lover here. The disappeared of Egypt, the jailed and flogged blasphemers of Saudi Arabia, the arbitrarily detained bloggers and journalists of China are being denied freedom of speech. It’s an insult to their ordeals that we equate them with shutting down Milo Yiannopoulos’s Twitter account. Over the past month, while travelling in north Africa, I was unable to access a host of mainstream news sites that had been blocked by national governments. That is censorship. This is not that complicated.

All that makes existence valuable to anyone depends on the enforcement of restraints upon the actions of other people John Stuart Mill

But it’s the “thin end of the wedge”, we’re told. If we start banning those whose views we don’t like, what next? In general, one should be suspicious of “what next?” arguments, because they assume that humans are incapable of behaving in calibrated ways that don’t inevitably lead to some future state of fascism. We could extend the right to platform and rally to all, but what next? Paedophile rallies? That’s obviously absurd, but it highlights the fact that there are limits, and they are broadly dictated by how much certain values are coded within society. The reason free speech proponents are not out there fighting to hear from child abusers or some radical Muslim clerics is because society or the law regulate the more unpalatable or illegal views away before we have to deal with them at Speakers’ Corner.

In On Liberty, John Stuart Mill, one of the great defenders of free speech, says a struggle always occurs between the competing demands of authority and liberty. He argues that we cannot have the latter without the former: “All that makes existence valuable to anyone depends on the enforcement of restraints upon the actions of other people. Some rules of conduct, therefore, must be imposed – by law in the first place, and by opinion on many things which are not fit subjects for the operation of law.”

But we’re far from that. Freedom of speech is no longer a value. It has become a loophole exploited with impunity by trolls, racists and ethnic cleansing advocates. They are aided by the group I call useful liberals – the “defend to the death your right to say it” folk. The writer Mari Uyehara calls them the “free speech grifters”, those “who flog PC culture as a singularly eminent threat to the freedom of expression”. To them, the “what next?” argument foresees apocalyptic harm that might befall liberal values. It cares much less about speech we can link to violence, or that which compromises the safety of others.

Last Sunday Tommy Robinson, the former leader of the English Defence League whose social media posts were cited repeatedly in the trial of the Finsbury Park terrorist Darren Osborne, gave the speech at Speakers’ Corner that would have been delivered by one of those denied entry – the Austrian Martin Sellner, a leader of the white supremacist group Generation Identity. Useful liberals have swallowed two freedom of speech myths whole: the redefinition of the term to encompass not only freedom from persecution but the right to a platform; and the delusion that freedom of speech is a neutral principle uncontaminated by history or social bias. There are hard choices here. Too often, those who should know better argue for the wrong ones. They fight to their deaths to defend the rights of Bachmann, Sellner and the other peddlers of hate – but not mine.

• Nesrine Malik is a Guardian columnist