The right to free speech is an essential ingredient for any modern democracy, but were it completely unlimited, it would not serve the complex needs of our society.

International human rights law recognises that the right to free speech is not an absolute right that trumps all others. As a society we are free to decide what we're willing to tolerate and where we need to draw the line to protect the rights of others.

Organisations like the Human Rights Law Centre are often involved in cases that explore where this line should be drawn. We've worked on various pieces of litigation seeking to help define the protections of free speech in Australia and when limitations are justified.

In one case, in which street preachers were prevented from leafleting in Adelaide’s Rundle Mall, we challenged the excessiveness of council by-laws that stifle political expression in public spaces. Recently we've taken legal action against the Melbourne City Council for not acting to ensure women can safely access an abortion clinic without being harassed or intimidated.

At first glance, these might seem contradictory. Nuanced positions are always susceptible to simplistic attacks, but such criticisms fail to recognise that the grey areas where competing rights clash need to be regularly surveyed, debated, and reconfigured for unique situations. Blanket rules just aren't suitable.

Passionate arguments that any restrictions on speech extinguish our freedom can be rousing and sound noble, but these claims are overly simplistic. They ignore that as a society we already place many reasonable restrictions on speech.

We cannot publish comments that would defame someone's reputation. We cannot hide behind a twisted notion of political expression to justify the production of child porn. We cannot engage in false advertising to sell goods. The reality is that no modern democracy has a complete guarantee of free speech.

Yes, it takes vigilance to ensure we don’t become complacent and let governments unnecessarily limit our rights because it's easy or convenient. The new move-on powers granted to Victoria Police are a good example of such overreach. But sometimes it’s fair to draw a line.

For example, do we think that hate groups should be allowed to disrupt the funerals of fallen soldiers in the name of free speech? In the United States, protest exclusion zones around funerals have been found to be constitutional; a legitimate limitation on free speech to protect parents and other loved ones targeted by extremists.

If America, the self-perceived bastion of free speech, can live with this compromise, Melburnians should be willing to embrace a similar arrangement of safe-access zones around abortion clinics.

The group that has besieged the Melbourne Fertility Clinic for decades has been known to physically block people from accessing health services. They thrust offensive pamphlets into the hands of four year olds, telling them that their mothers are murdering their little brother or sister. They force pictures of bloodied foetuses on women and staff entering the clinic.

You would think common decency would prevent this type of behaviour, but regrettably that’s not always the case. At times the law is required to intervene to stop bullying behaviour and ensure that basic community standards are met in practice.

The issue is at the core of the current debate around our racial vilification laws. Attorney-General George Brandis’ insistence that people have a right to express bigotry ignores the tangible harm that can flow from racial vilification.

The proposed amendments to the Racial Discrimination Act that Brandis has released for community consultation water-down the laws that provide protection against harmful racial hate speech.

The most concerning aspect of the proposals is an extraordinarily expanded exemption granted to almost all forms of public discourse. This means that anyone with access to a megaphone, blog or twitter account can incite racial hatred, rendering the prohibition meaningless.

By all means, let’s have a sensible, mature, debate about exactly where and how to draw the line, but we must remember that free speech is not a license to harm others with impunity.