"No reasonable person can object to the protection of rights," the anti-charter academic Helen Irving wrote recently. "Most are concerned, rather, about the best means of protecting rights. The central issue is whether the best means revolves around judicial review." That polite formulation needs to be sharply questioned. I'm sure it's true for Professor Irving. But most campaigners against bills and charters aren't driven by abstract constitutional concerns. They are out on the hustings, working the corridors of Canberra and polishing newspaper articles because they do object to the protection of rights. That is the point. This is not a battle about ways and means. This is a contest over outcomes. Read the elegant essays of Senator George Brandis, note his appeals to British ways now outmoded even in Britain, and by the end you know he and his party are not on the side of rights. They are on the side of lobbying, influence and power. And so are large chunks of the Labor Party. Bob Carr, the man who loves so much about America, American history and the Democratic Party, loathes the First Amendment that makes America what it is. He wants nothing like it here. Carr is not alone in Labor. No national leader of the party has put any skin in the game since Bert Evatt departed already a little unhinged for the Supreme Court of NSW in 1960. Labor waves its human rights banner aloft as proof of its decency and good intentions. But when it comes to it, securing rights has been a second- or third-order issue.

So how is it to be managed? Referendums are out since the debacle of 1988. What an unexceptional list of rights were on offer to the people of Australia then. As Professor Irving would ask: what reasonable person could object to them? Yet all three proposals - to extend trial by jury, confirm freedom of religion and make state governments pay a fair price for properties they acquire - were lost overwhelmingly. The score is now 44 losses from 52 attempts to change the constitution by referendum. The lesson we are supposed to draw from that is that Australians hate change, or at least hate to change the constitution. That's not untrue. But the deeper lesson is that Australians - contrary to our larrikin myths - are people with a deep respect for authority. We move only when we're told. We straddle the great philosophical divide that runs across North America: we are more Canadian than Yankee, not so dedicated to "life, liberty and the pursuit of happiness" and more at home with the "peace, order and good government" that is supposed to reign north of the 45th parallel. The US finds it so much easier to change its constitution not only because the machinery of the thing is simpler, but because Americans are more used to making up their own minds. Australians require their leaders to speak as one before we will make a change. This isn't a commitment to the constitution but a commitment to authority.

No one should doubt how hard it is to embed human rights in constitutions. Rock solid guarantees of that kind only ever seem to follow national upheavals. Look at the list: the US after a war of independence; France after a revolution (and that one didn't last); Europe after World War II; and South Africa after the long nightmare of apartheid. Australia lost its chance as the constitution was being drafted. Whether we know it or not, we are still picking around in the wreckage of one February day at the Australasian Federal Convention in Melbourne in 1898. Apart from anything, it was staggeringly hot: 40 degrees in the shade. As fires raged through the Grampians and smoke obscured the sun, Australia's best hopes of a bill of rights were burnt to a crisp. Little fuss had been made at earlier conventions of the idea of incorporating into our constitution the "equal protection of the laws" established in the 14th amendment of the US constitution. But its enemies, led by Isaac Isaacs, were waiting to pounce in Melbourne. It's a strange reflection that the leaders of the contest that day were Australian sons of persecuted peoples: Isaacs, the brilliant, tedious, dogmatic child of a Polish tailor, and Richard O'Connor, the charming son of an Irish librarian. The 42 delegates growled and sniped for an hour, broke for lunch and came back to the Legislative Assembly of the Melbourne Parliament to shred the rights initiative in less than 20 minutes. Almost every trace of the 14th amendment was wiped from the Australian draft by a final vote of 19 to 23.

According to the great myth that has grown up about the decision, this was the point at which an emerging Australia rejected American ways and stuck to its British guns, turning its back on the allure of constitutional protection of rights in order to stick with the common law and responsible government. It has been billed ever since as a nation-making moment. But this is a judicial invention. Read the transcript of that day's debates and you find no such high-flown considerations in the air. No hymns were sung to British ways. Not even the most conservative delegate - stand up if you can after a long lunch, Sir George Reid - attacked the theory of allowing courts to set limits to the exercise of government power. The delegates were practical men. They did not vote against rights but against these rights. For, as Isaacs put it so bluntly, their original object in America was "to protect the blacks" and in Australia they would "protect Chinamen in the same way". The delegates' vote was not about preserving British values down under, but the birth of a white man's federation. What's the point of this history excursion? To bell another cat. The rejection of legal protections for human rights is not in the DNA of this nation. The fight is worth continuing. Success is not impossible. Alas, what is in our DNA is a high anxiety about extending rights to those Sir John Forrest called during that 1898 debate "coloured persons". Not for the last time, we chose between race and rights, and the price of the choice was high. Even today, Australia doesn't have what O'Connor called "a guarantee for all time for the citizens of the Commonwealth that they shall be treated according to what we recognise to be the principles of justice and equality".

In 2010 there is nothing in law to stop Western Australia closing its iron mines to Chinese workers. And the politics of rights protection continues to be - what is the polite word? - complicated by the fact that those who most obviously need protection aren't named McClelland or Evans or Ruddock but Haneef, Al Kateb and Ul-Haque. Isaacs lived a very long time: long enough to be our first native-born governor-general, to watch Kristallnacht from the distance of Melbourne, to witness Australia's appalling response in the late 1930s to Jewish refugees, and to see Auschwitz emptied. I wonder if at any time in his late life he reflected on O'Connor's wise words on February 8, 1898, about the role the law might play in protecting us all from the occasional madness that sweeps nations. "We are making a constitution to endure, practically speaking, for all time," O'Connor said. "We do not know when some wave of popular feeling may lead a majority in the Parliament of a state to commit an injustice by passing a law that would deprive citizens of life, liberty, or property without due process of law. If no state does anything of the kind, there will be no harm in this provision, but it is only right that this protection should be given to every citizen of the Commonwealth." It wasn't then and it's still not there now.

This is an edited version of the 2010 Human Rights Oration delivered yesterday at Federation Square, Melbourne.