Once upon a time, I believed in the right to be a bigot. I was a young, middle-class, private school educated white male living in Ireland, and I firmly believed in my right to offend.

At university, I was in more than one debating society. My idea of fun was to put on a tuxedo and solve the problems of the world one speech at a time. In all honesty, I was pretty good at it. With 15 minutes worth of prep and a seven minute speech, I could convince you that black was white or white was black.

Arguing that black is white is what got commentator Andrew Bolt into a spot of bother in 2010, when he implied light-skinned people who identified as Aboriginal did so for personal gain. Bolt was found guilty of a breach of section 18C of the Racial Discrimination Act, and that conviction has set a train in motion resulting in George Brandis, the Commonwealth attorney general, now declaring that there is a right to be a bigot.

I can’t speak for everyone but in my experience, many families have an ageing relative who pops up with the most inappropriate pronouncements at the Passover Seder or Christmas table. But having such opinions is not the same as having the right to broadcast those views. And that fundamental truth is revealed in the attitude of the attorney general towards Holocaust denial.

The Jewish community has been repeatedly assured that Holocaust denial would remain unlawful under the proposed exposure draft of 18C. Why? If this reform is about free speech, why is the government introducing an offence of racial vilification? Why is it at pains to stress that Holocaust denial remains beyond the pale? If this government believes in the marketplace of ideas, then why not open up that market? The cynic in me would argue that reassurances about Holocaust denial are an attempt to silence the Jewish community in this debate.

The thing that proponents of this marketplace of ideas ignore, but which minorities should never forget, is that speech has a context.

Take for example a mate of mine, who was recently on the train with his primary school aged kids. A drunk on the train starts having a go for no reason; he probably would have had a problem whether they were black, white, Asian, or Muslim. But on this occasion, they were Jewish. So the guy starts on about "rich Jews", and uses this trope to yell offensive nonsense at a seven-year-old boy. It's ghoulish, but it's also especially offensive because these insults have a history which cannot be simply brushed aside.

When someone makes monkey noises directed at a football player, we all know what it means. When someone making similar offensive noises is a young girl and the subject of the abuse is Adam Goodes, the Australian of the Year, it may attract more attention. But either way, portraying people as monkeys has a history and a specific context – which is why modern day politicians can't act as if the White Australia policy never happened and destroyed lives.

Bearing that history in mind, we should not forget how long it took before Gough Whitlam poured a fist full of sand through the hands of Vincent Lingiari. Before Paul Kelly sang about it. Before Nicky Winmar raised his jersey, taking a stand against racist comments. Before Kevin Rudd apologised to Indigenous people.

Kevin Rudd after his landmark apology. Photograph: AFP/Getty Images Photograph: AFP/AFP/Getty Images

Likewise, there is a context to 18C. There is a context when people seek to belittle the fact that European settlement was based on the offensive legal pretext that the Australia was “unsettled” – that it was terra nullius.

There is a context when the privileged seek to deny the Stolen Generations by implying that being light skinned makes someone less Aboriginal. That context can mean that a seemingly inoffensive phrase is, in fact, offensive. Given that history and context, it is wrong to privilege the rights of bigots over the rights of minorities, especially since there is no evidence that 18C was having a chilling effect on the debate about race in Australia. On the other hand, repealing 18C makes a statement: that there is a right to offend and that the determining what is offensive is a matter for the majority, for the privileged. Those priorities seem skewed to me.

I don’t speak for the Jewish community, or any minority group for that matter. But I know that the current exposure draft to amend 18C does not acknowledge the context within which hate speech occurs.

Everyone who has ever felt belittled by speech, and who is capable of empathy with Australia’s First Peoples, ought to draw on that experience and oppose this amendment – even if we don’t think this change will affect the Irish, the Jews, the Lebanese or the Muslims.

This is about more than just Holocaust denial. And it’s not every day that you get to write that sentence.