PEN's Free Voices lecture series was designed to provide a space for the diverse perspectives of a range of writers to explore the concepts of "freedom to read, freedom to write, freedom to speak." These are the principles that drive PEN International, whose members throughout the world work to free imprisoned writers and journalists and to heighten awareness of concerns about free speech, censorship and repression. In his 2013 PEN Free Voices Lecture, broadcaster and writer Waleed Aly agues that if considerations of power are excluded from our debates on free speech, the debates are destined to be distorted.

If I were giving this speech about two-and-a-half thousand years ago, there might well have been an empty chair beside me representing Socrates. To be sure, the window for this would have been small - he was tried, convicted and executed within a year - but on principle he seems an ideal candidate. His offences against Athenian society were to fail to acknowledge the city's gods, and to introduce new deities. In short, his were offences of speech.

Socrates was a provocateur. He revelled in his dissent. His self-proclaimed function was to be a nuisance - to sting the city with his philosophical questions. He was contemptuous of democracy, believing that the best policies were the product of the wisdom, insight, knowledge and competence possessed by an elite few.

This was no small claim in Socrates' Athens, which of course prided itself on the audacity of its democratic experiment. In this, it stood in stark contrast with Sparta and Crete. Naturally, Socrates went on to express his admiration for those societies. Then there was the fact that one of his students, Critias, was part of a group, backed by Sparta, that briefly seized power in Athens, running a brutal, undemocratic government. Eventually these interlopers were vanquished and democracy restored. Socrates was tried soon after.

There are seditious overtones to all this. Still, his execution was significant precisely because Athens valued free speech and dissent. And yet, a majority of voting Athenians resolved to put him to death for his words. No doubt, his case is more complicated than that of your average dissident, but it stands as a symbol of the fragility of free speech. Even in societies that value it, vigilance is important.

It is therefore a good thing that free speech is never far from the surface in Australian political debate. But that doesn't make its application a straightforward matter. Recently that debate has turned to the question of racial vilification laws. This is partly because the Gillard government has declared its plans to pass new legislation consolidating all anti-discrimination legislation that would make some changes to the racial vilification laws we currently have. But it is also because of a recent Federal Court judgment against News Limited columnist Andrew Bolt that found he had racially vilified some Indigenous complainants.

The government's legislative plans now seem largely theoretical. Recently, Attorney-General Mark Dreyfus made clear that this legislation would not be rushed through, and has refused to give a timeline even for its final drafting, much less its introduction into parliament. Clearly there is no rush. Given there are now very few sitting days before the September election, that the government is almost certain to lose that election, and that the Opposition has made plain its rejection of that legislation, there is now little practical point in considering it in detail. But the issue is still a live one because the Opposition has also signalled its intention to amend our current racial vilification laws, to loosen some of the restrictions on speech that currently apply. So now the most politically relevant question becomes whether or not the legislation we currently have is an odious restriction that we should abandon.

I'm interested in answering that question, not so much for its own sake, but because it gives us an opportunity to think deeply and systematically about the whole concept of free speech. Frankly, we don't do this enough. We cite it as a bedrock of our politics and society, but perhaps because we tend to agree it is so fundamental, we rarely engage with what it really means, on what basis we should guarantee it, and therefore on what basis it can legitimately be restricted.

And in discussing this, I want to emphasise an extremely important part of the equation that is almost never meaningfully discussed in our public discourse: power. Whatever your position on racial vilification laws, it must account for the way in which power is distributed in our society. To ignore power is to ignore the most basic social context in which we are having this debate. It condemns us to a deeply impoverished consideration of free speech. And that matters because, if we are approach free speech with vigilance, we need the ability to tell when it is really under threat, and when it is not.

Why there is no such thing as "free speech"

Already, I'm assuming that not every restriction on free speech constitutes a threat. That, I think, should be uncontroversial. Even the most liberal approaches to free speech - such as that of the great liberal thinker John Stuart Mill - accepts limitations are necessary. This is probably why every single society in human history has adopted such limits. Defamation law - which is a thriving legal area in the United States, even with its shiny first amendment - is the most obvious example of this. Another might be speech that incites to violence, or the publication of child pornography, since such speech does direct harm to others and so cannot be protected by the doctrines of liberalism. Free speech arguments are really arguments about the extent to which speech should properly be restricted, and which means are the most appropriate for doing so. Indeed, Mill himself concedes some restrictions are desirable and important.

"All that makes existence valuable to anyone depends on the enforcement of restraints upon the actions of other people. Some rules of conduct, therefore, must be imposed - by law in the first place, and by opinion on many things which are not fit subjects for the operation of law."

There are two ideas here. First, that where some action should be restrained, we must ask whether the law is a necessary method of doing this. Second, that other forms of restraint and control are available, and specifically that we may be restrained by social forces. Society can - and indeed, should - restrain us even where the law doesn't. It can censor us. And this much, simply cannot be denied.

In any society, or even any social setting, some statements will be riskier than others. Some will cause the speaker to be celebrated, others will have them ostracised. And as social beings, we typically weigh up the likely consequences before we open our mouths. That's why we adjust what we say - and especially the tone of it - to suit our audience. We are forever asking ourselves, "What reaction am I looking for, and what is the best thing to say to get it?"

In my broadcast work, I interview politicians all the time. The answers they give me on air will be different to what they tell me in private. Off air they will be more prepared to admit party weaknesses, they will speak more frankly about social issues. That is, they will be less restrained, because they know the consequences of their speech are unlikely to be dire in the same way they might be if they spoke so frankly to a broadcast audience.

If you're honest about yourself, you'll probably find you're much more diplomatic to your boss, than you are about your boss. This isn't necessarily a character flaw. In fact, it's a good sign of our social intelligence. People who barrel on without any regard for social sensitivities or the social consequences of their speech are exhibiting one of the classic traits of autism, or in extremis, sociopathy.

That doesn't mean we never say anything controversial. We might decide what we have to say is important enough to be worth the social cost of being the target of moral outrage, or we might believe that there is some social benefit to being a controversialist. The point, though, is that social pressure will almost always be a factor in our decision.

So let us grind this out, beginning with a trite observation: that, in some sense, we are always free to speak. Even if our speech will land us in prison or before an executioner, there is nothing anyone can really do to stop us saying whatever it is we want to say. The only question is whether or not we are prepared to bear the costs of our statements. We can never escape those costs. Even an imaginary society with no legal restrictions on speech at all will still have mechanisms for making certain speech costly in one way or another. Only when speech is entirely meaningless to the audience can it have no cost, benefit or consequence (which is why swearing in a foreign language is nowhere near as crude as swearing in the dominant one).

Put simply, there is simply no such thing as free speech. There are only different costs. When we say we support free speech, we're actually saying something very imprecise. What we really mean is that the costs of speech should not be imposed by the State, and where possible, social pressure should decide.

As general rules go, that's a pretty good one. Throughout human history, the State has repeatedly demonstrated an awesome ability, and preparedness to kill or imprison people who say things it doesn't like. State intervention is generally worse than social intervention because it is typically more brutal and heavy-handed. Certainly, social opprobrium can have extreme consequences. We can easily imagine how, for example, the cost of saying "I am gay" in a particularly rough school yard could have could be extremely high. The consequence could be relentless bullying, perhaps even violence. It could drive someone to self-harm or even suicide. Weighing this up, you might choose not to say anything.

There is simply no such thing as free speech. There are only different costs.

In fact, you'd have to say a social cost like that is a far bigger restriction on free speech than, say, having to pay a fine for something you say. But it would be perverse to deal with this by imposing some kind of legal penalty for admitting your sexuality. The main problem is the social stigma behind the bullying. That won't go away by fining on the victims of bullying. In fact, that would probably amplify the stigma. And that is often the problem with State-regulation of speech. It doesn't replace the social cost of speech, and it doesn't reduce its chilling effects. It just compounds them.

So, a presumption against State intervention is a good idea. But it's just that: a presumption, not a rule that should be applied in every case. Only rarely should that be overturned. The trick is to figure out when those cases should be. We therefore need to identify - as a matter of principle - how we decide when State regulation of speech is a good idea.

Imbalance of social power

Let's start with liberalism. Mill would like to see offensive arguments left to society to sort out. Much as we might rely on the free market to separate the quality products from the shonky, we should allow the marketplace of ideas to put offensive speech in its place, or alternatively, discover some hidden merit in them. It's a neat idea, and for the most part it's persuasive. But there's a problem here.

Leaving things to social regulation means we're relying on social power. And the thing about social power is that it's not distributed evenly, or fairly. There are those with social power, and there are those without it. The social regulation of speech therefore places regulation in the hands of the powerful. And while it is true that not all powerful people will use their social power in their own interests and against the interests of the powerless, it is also surely true that plenty of them will. Social regulation becomes largely the regulation of the powerless by the powerful.

This, as it happens, is one of the major reasons we are suspicious of government regulation. We don't trust the political class to resist silencing voices that speak against their interests. But why should those with other forms of power be trusted so much more? Are they somehow free from self-interest? Are they incapable of abusing their power, of being corrupt dishonest, bigoted or otherwise odious influences on society? If not, we're confronted with the very real prospect that the social regulation of speech will mean that the most costly speech is that which doesn't serve the interests of the powerful: those in our society with economic, social and cultural power.

All this is utterly central to the issue of racial vilification. In fact, it is central to the whole politics of racism. Racism is such an odious social and political force, not simply because it is a baseless prejudice, but because it mobilises hatred against those who are powerless to protect themselves. This is why what is so often called "reverse racism" - where those who are typically thought to be the victims of racism begin racially stereotyping their assailants - is simply nowhere near as socially or politically significant.

The grotesquely anti-Semitic cartoons published in Nazi publications in 1930s Germany were far more contemptible than if Jewish publications of the same period started similarly lampooning Germanic people. Not because Jews have a special licence for prejudice, or because there is something inherently worse about anti-Semitic racism than anti-Germanic racism as a matter of abstract principle. But because it is dangerous and frightening when one social group with power and influence demonises another without them. Disempowered people lampoon their overlords as a means of seeking equality. Empowered people vilify their underlings as a means of subjugation. The two are simply not equivalent.

It's the same reason women can say things about men that men could never say in reverse. Men, frankly, have no place complaining of sexism in these circumstances and challenging for the right to demean women. Not while they dominate our corporate boardrooms, or our parliament. Not while they're earning significantly more than women. Not while they claim a disproportionate presence in (particularly commercial) media. Not while their career and social prospects are far less limited by their physical appearances. And not while they are overwhelmingly perpetrators, rather than victims, of inter-gender violence. In that context, we cannot dismiss calls for affirmative action as indistinguishable from discrimination. The intentions, as well as the consequences, are completely different.

This sort of reasoning could easily apply to a range of circumstances: gay and straight people, or rich and poor people, for example. But for present purposes, let us consider what it means for the issue of racial vilification. If, for example, a white, middle-class male with a mainstream media platform embarks on a racist tirade against black people, relying on questionable information and a septic reservoir of stereotypes to build the case he is making, he is doing this in a particular social context. If he's a broadcaster, his manager is almost certainly white, and probably male. If he's a columnist, the same might be said for his editor. Indeed, the same might be said all the way up the chain until we reach the most influential person in the organisation, whether that's a managing director, a major shareholder, an editor-in-chief or indeed a newspaper proprietor. Every day these influential people will make decisions about what to place on the national agenda. Those decisions will inevitably reflect their priorities; their sense of what is really interesting and important. And those priorities will necessarily be informed by their own social circumstances and experience. In this way the society presented to the public will be to some extent the society as it looks from that privileged vantage point.

Social power is not distributed evenly or fairly. There are those with social power, and there are those without it. The social regulation of speech therefore places regulation in the hands of the powerful.

Taking all this together, we can safely say the person responsible for the racist tirade belongs to about the most socially powerful subset there is in Australian society. Moreover, he is among the more socially powerful members of that subset given the media platform he commands.

Consider, now, his victims. They almost certainly have no institutional power in the mainstream media. They are unlikely to have any influential representatives in the editorial meetings that determine what the public conversation of the day will be. Ultimately they have little to no access to mainstream media platforms - certainly not of right, and certainly not to the same extent as the white middle-class male that has just assailed them.

Here, it is worth noting two things. First, that it is possible that some individuals among the vilified group will have the skills, the connections and the access to make their voice heard. But those voices will always be buried by the onslaught of privileged voices. Their existence does not put the discourse on a level playing field at all. Second, there's no doubt that some among the empowered social subset will be sympathetic to the plight of the vilified and perhaps even come to their defence. But this must not be mistaken for an equalisation of social power. Here, the vilified group remains a kind of supplicant, relying on the discretionary magnanimity of its social superiors. It is not mounting its own argument on its own behalf informed by its own experiences and intimate knowledge of itself. It is hoping for the benefit of arguments made by those who probably cannot truly understand the nuances of its social existence. The bottom line is that so many of the public conversations we have about minorities, and especially racial and ethnic minorities, take place as though those minorities are not in the room. They are variously accused, prosecuted, defended, convicted, exonerated and deconstructed, but they are very rarely heard.

If free speech is meant to be analogous to the free market, if bad ideas are to be vanquished by good ones in the contest of ideas, then what happens where that contest scarcely exists? Really, it's like an abuse of market power: a kind of market distortion. There is at the very least a case to be made for regulating speech in these circumstances to ensure that the discourse of the socially empowered is held accountable in some way. Not because it is offensive or hurts people's feelings, and not because I think the law is always an effective instrument for curing society's bigotry (in fact, I don't think society can ever be free from bigotry). But rather because it serves to marginalise vilified minorities from the public conversation, and from participating in our democratic life. In the long run, that risks silencing such communities, thereby reducing the range of contesting opinions. That serves almost no principle on which the case for free speech is based.

But it's a difficult balance. Ideally, we would avoid outlawing certain opinions per se. There is something inherently objectionable about making it legally impossible to state a given position - even an offensive one. But it is possible to place requirements on how certain inflammatory arguments should be put. We can require, for example, that they are put honestly: that they are not full of fabrications or gross distortions. We can also require that, particularly in the case of dangerously inflammatory ideas, that they are conducted with a certain tone that reduces the likelihood of some manner of social explosion. Such limits would not be moralistic or paternalistic, which means they are still broadly consistent with the liberal structure of our society.

I accept that, as a practical matter, all this might make certain arguments extremely difficult to run. Holocaust denial comes to mind. But it does leave open the theoretical possibility of someone arguing a Holocaust denial position, provided they can do so without factual fabrication or distortion and in a non-incendiary manner. And if that is impossible to do, we are entitled to ask why such arguments are of such a benefit to society that the right to air them must trump whatever rights a vilified racial group should otherwise enjoy.

Lawful offense and the Racial Discrimination Act

With that in mind, let us consider the recently controversial racial vilification provision in the Racial Discrimination Act. The battle centres on section 18C, which says makes public speech unlawful if it "is reasonably likely in all the circumstances to offend, insult, humiliate or intimidate another person or group of people" because of their "race, colour or national or ethnic origin."

Shadow Attorney-General George Brandis has frequently complained that this makes it unlawful for people to say offensive things. And that is true in certain circumstances, but it's misleading to leave it at that. For a fuller picture, we must read the next section, 18D, which specifies exactly what sort of statements are not unlawful. Those include:

anything said or done reasonably and in good faith: (a) in the performance, exhibition or distribution of an artistic work; (b) in the course of any statement, publication, discussion or debate made or held for any genuine academic, artistic or scientific purpose or any other genuine purpose in the public interest; or (c) in making or publishing: (i) a fair and accurate report of any event or matter of public interest; or (ii) a fair comment on any event or matter of public interest if the comment is an expression of a genuine belief held by the person making the comment.

So let's be clear about this. The law allows you to insult, humiliate, intimidate and of course offend a racial minority, on racial grounds, if it's a fair comment on a matter of public interest based on a genuinely held belief. The requirements that the comment is "done reasonably and in good faith" are, I would argue, limitations of tone rather than content. A racist tirade that fabricates "facts" and misrepresents the people it is vilifying is unlikely to qualify. An uncomfortably racist argument that is nonetheless honest and intellectually defensible is far more likely to be legal, no matter how offensive it is. It does not make it a crime merely to offend people.

The restriction here is not on the scope of the debate, or indeed on the level of offence. The law simply requires that if you're going to publish racist arguments that impugn disempowered people, you must do so in a manner that is honest and fair. The argument itself doesn't have to be fair. But the way in which it is put together and expressed, does. Certainly it imposes a greater burden of diligence on the speaker. But given the potential social and political dangers of racism, there's a strong case this is no bad thing. If it enforces anything, it is a norm of civility in cases with high risks of combustion.

It is simply untrue to insist, as George Brandis does, that Australians "are not free to make critical or unpopular remarks in the course of ordinary political exchange." We are. There is no provision of the Racial Discrimination Act or any other Act that outlaws "critical or unpopular remarks." For better and worse, we do it constantly. Brandis was speaking in response to the Federal Court judgment against Andrew Bolt for racial discrimination, but even that further context doesn't make Brandis' critique any truer.

In that case, Bolt had published two articles suggesting that a group of fair-skinned Aborigines were emphasising their Aboriginality over their European lineage for some kind of professional gain. This provides a classic basis for a defamation claim, but the offended parties pursued a racial discrimination claim because there was something bigger at stake here than their own personal reputations. This was a privileged white person attempting somehow to police the extent to which people with Indigenous heritage should indentify as such. This is of amplified significance when we consider the history of white people attempting to determine what the identities of black people should be; whether it be attempts in Australia to raise "half-caste" children as white, or the Southern American "one drop rule" that deemed anyone with even a single drop of "black blood" wouldn't be treated as white.

Liberalism wants to free the individual from the constraints of society's sensitivities. And yet, to the extent it leaves these things to the regulation of social pressure, it cannot help but privilege those who already have power.

For Bolt, the argument may have been about the motivations of individuals, but for the complainants it was about (in the Court's language) the freedom "to fully identify with their race without fear of public disdain or loss of esteem for so identifying." That is to say, it was about their freedom to declare themselves Aboriginal without bearing a significant social cost - a freedom they have not always enjoyed. Theirs was a free speech claim of sorts, too. Just not one our public discourse trains us to recognise.

But crucially, the Court did not simply say Bolt's argument was too critical or unpopular. It specifically said the offending articles "contained errors of fact, distortions of the truth and inflammatory and provocative language," which indicated that Bolt was trying to be "destructive of racial tolerance." That is, his writing was not reasonable, in good faith or even honest. Brandis has been at pains to make clear he is not criticising the judge who "was merely applying the word of the statute to the facts of the case before him." But if Brandis accepts the Court's findings about Bolt's error-ridden, distorting work, he simply cannot pretend the case was about "critical or unpopular remarks in the course of political exchange."

And surely it is an exaggeration to cite this as proof that Australian commentators are not free. The sanction in Bolt's case was that his newspaper would have to print an apology or correction, and refrain from publishing anything substantially similar - presumably that means something similarly factually suspect - in future. Frankly, newspapers should refrain from publishing such material at all times. That's the function of a newspaper. The bigger cost here is really the social cost of being named as someone guilty of racial discrimination. This is hardly heavy-handed State intervention. It is a principally social cost, triggered by a legal finding. All that considered, the question really is why Brandis thinks well-paid, socially powerful columnists should be free to write factually incorrect, racially vilifying material against Indigenous Australians without even the slightest legal sanction.

The answer to that question surely lies in the concept of power. Brandis' response does not account for it at all. The only power he sees here is the Court's power to censure Bolt, albeit lightly. That is, in some ways, a classically liberal point of view. Brandis' position on this case throws into stark relief the limits of liberalism - namely, that it is overwhelmingly blind to power in its approach to freedom.

Indeed, that is perhaps liberalism's greatest conundrum. It is a philosophy born of a radical spirit. It exists precisely because it doesn't trust majoritarian authority. It wants to free the individual from the constraints of society's sensitivities. And yet, to the extent it leaves these things to the regulation of social pressure, it cannot help but privilege those who already have power. In this way, and especially in societies that are already liberal, it acts in a surprisingly conservative way, because it reinforces the status quo, which is probably why the conservative side of politics in Western democracies tends to be more avowedly liberal than their more typically socialist, progressive opponents. This is why conservative parties usually end up promoting policies that work in the interests of the elite, even while being liberal.

Free speech worth fighting for

I believe free speech is one of the most fundamental features of a plural, open, democratic society like ours. But it's not the only one. The equality of citizens is another. Equal opportunity of democratic participation is another still. I don't think it is good enough simply to declare the supremacy of free speech over all other social interests as though it is some unproblematic truism. I believe we must always be asking that most fundamental question: what is the point of free speech? Is it its own end? Or does it serve some other purpose? Put another way, should we organise our society to maximise free speech, or calibrate free speech to serve society in some way or other?

It's for that reason that I don't think racial vilification laws are some kind of blot on our free speech landscape. In fact, I think they serve an important democratic function. In saying this, I'm alive to the dangers. I understand that it is easy for the State to silence dissent on the pretext that it undermines social cohesion when what they really mean is that it undermines State authority. But our vilification laws are not designed to give the State power to lock up dissidents - and in any event, I can't think of a single case of this happening. They are used rarely, and provide mainly for civil action. Remedies are typically low-cost gestures like publishing statements of acknowledgment.

Of course, the mere existence of vilification laws on the books does not undo the power disparities in society, and I accept that even those who have civil rights under this legislation will often lack the means to enforce them. But that only suggests the effect on limiting speech will be minimal. What effect there is will be some form of "chilling", where self-censorship operates. That is, of course, a concern. But it must be weighed against the fact that the victims of vilification rarely have access to any significant platform for public speech at all. They are censored by social and economic barriers that are far less voluntary, and far more comprehensive than a chilling effect.

Ultimately, we mislead ourselves if we think we can examine free speech in any meaningful way without paying attention to its relationship with power. In fact, being mindful of power dynamics should only amplify the importance of free speech as a foundational issue, and make clear to us the circumstances in which is it so essential to fight for it.

It is for precisely this reason that I regard the PEN message as fundamentally important. It is, almost quintessentially, about defending the powerless from the tyranny of the powerful. The poets, essayists and novelists who inspire it are overwhelmingly those persecuted, imprisoned or exiled for their dissent. It has been a voice for the powerless in - and often excluded from - our society, such as asylum seekers and Indigenous Australians. These are laudable aims and at their core, rarely if ever discussed, is the concept of power.

Waleed Aly is the presenter of Radio National's Drive program and lectures in politics at Monash University, where he also works within the Global Terrorism Research Centre. He is the author of People Like Us and What's Right: The Future of Conservatism in Australia.