Martin Sellner is a nasty piece of work. The Austrian is a leader of the Generation Identity white nationalist movement and of Defend Europe, an organisation that tries to stop NGO boats from saving migrants drowning in the Med. His partner, the American Brittany Pettibone, is a writer and vlogger who sees immigration as “white genocide”. Lauren Southern is a Canadian journalist who moves in far-right circles. Hitler, according to Southern, “was just an SJW [social justice warrior] who happened to get freaky amounts of power”.

Over the past two weeks, all three have been barred from entering Britain. Their views, they were told, were unacceptable and their presence in Britain not “conducive to the public good”.

Sellner, Pettibone and Southern are all odious characters. I loathe their anti-immigrant rhetoric. I despair of their tirades against Muslims. But I also think that all three have the right to be as loathsome and obnoxious as they wish to be. Even scumbags should not be barred for thought crimes.

Anti-racists in particular should be wary of such bans. Censorious laws that some applaud when applied to the far right inevitably get turned on to the left and anti-racists. The 1965 Race Relations Act introduced Britain’s first legal ban on the incitement of racial hatred. Among the first people convicted under its provisions was the black power activist Michael X.

The 1936 Public Order Act, brought in to control Oswald Mosley’s fascists, became used after the war to target trade unionists – most notoriously during the miners’ strike of 1984-85 – and anti-fascist demonstrations in the 1970s and 1980s.

If we allow the state to define the limits of acceptable speech, it will not just be speech to which we object that gets curtailed.

• Kenan Malik is an Observer columnist