When Michael Kirby first went to the High Court in 1996, Justice Michael McHugh gave him a little talk: “Michael, there’s one thing you must understand about this place and it is the importance of the number four.”

McHugh lifted four fingers as if to emphasise the point. With a court of seven members, you need four on the same ticket to get anything done.

As a prominent dissenter, Kirby knows about this too well. “If you have three, that’s good and may herald a new dawn,” he says, “but until you get four you don’t have the majority and therefore you don’t lay down the principle of law that is binding.”

One thing Kirby understands after 13 years as a High Court justice is that the crop of judges at the top of our judicial food chain is not much interested in human rights or what can be learnt from the jurisprudence of comparable democracies.

Kirby proffers this as a possible explanation for why Yale Law School dropped senior judges from Australia from the invitation list to its annual high-powered constitutional conference where issues of international comparative law and in particular the international law of human rights are thrashed out.

“They probably came to the conclusion that none of the judges of the High Court of Australia were very interested in those matters,” Kirby says.

In saying that, he knows full well the current justices will be wagging their fingers in disapproval. Advice from retired members of the court is met with disdain. Justice Kenneth Hayne dismissively used to say, “a member of the mothball fleet has been firing salvos”.

Not that Australia’s most public former judge was lacking circumspection when he spoke to The Saturday Paper. He deftly skirted answers to the question about why the court doesn’t assert its judicial power and declare unconstitutional laws that see refugees and asylum seekers detained indefinitely offshore.

Rather than jealously guarding its powers under the constitution, it appears the court is more tangled in arguments about the extent to which these refugees are “detained” and therefore punished, and if so whether this is the doing of other sovereign governments in some sort of elaborate fiction that requires Australia to be constantly washing its grimy hands while looking in the other direction.

Kirby is troubled that the Uluru Statement from the Heart is misinterpreted and distorted. He thought it surprising that former prime minister Malcolm Turnbull rejected the statement out of hand.

The outcome is that the judicial power of detention and punishment has steadily been usurped by the political arms of government, so much so that in our name people are indefinitely deprived of their freedom without charge or evidence of wrongdoing.

Virginia Bell, Stephen Gageler and Michelle Gordon (Mrs Hayne) are three serving High Court judges who to varying degrees have indicated they see through the fiction and that in their view the piece of string that represents indeterminate “processing” of refugees by the executive may be running out of length.

As if we need reminding, Kirby adds, “the principal hasn’t gathered up the magic number four”. That was the closest we got to “a salvo from the mothball fleet”.

Kirby retired from the court 10 years ago at the Commonwealth’s statutory age of senility, 70, yet he hasn’t learnt to slow down. He gives hundreds of speeches a year, travels incessantly, launches books and festivals. A conference isn’t a conference unless Michael Kirby is there.

His rooms opposite Parliament House in Sydney’s Macquarie Street are laden with photos, memorabilia, plaudits, awards, inscriptions and hosannas. There are snaps of Kirby with the royals, with George “Dubya” Bush, the emperor of Japan, with leaders of the United Nations and the free and not-so-free world. Dinners, research centres and law schools are named in his honour.

Just before we talked he had been in Melbourne giving the annual oration for Diversity Council Australia. Moments earlier he was back from Budapest and a human rights talk with judges and lawyers from Hungary, Poland and Romania “who are very much influenced by the Russian Federation and its approach to human rights, which isn’t particularly favourable”.

Behind me in the interviewing queue that day are a special representative of the UN wanting to discuss war crimes against humanity in Cambodia and a woman keen to explore Kirby’s early musical memories – operatic soprano Dame Joan Hammond and Evie Hayes from Annie Get Your Gun are on his list for starters.

His phone rings numerous times each day with plaintive pleas from people beset with “miscarriages of justice”. He quietly explains he is no longer a judge, a solicitor or a barrister and there’s not much he can do to rescue them from their plight. “It’s the problem of getting access to our system of justice and law … it is a very expensive system.”

There are a cornucopia of issues that occupy his thoughts – an Indigenous voice to parliament, religious freedom, minorities and a bill or charter of rights.

Kirby is troubled that the Uluru Statement from the Heart is misinterpreted and distorted. He thought it surprising that former prime minister Malcolm Turnbull rejected the statement out of hand. “He didn’t even go through the paces that politicians of the current generation are really expert in, and saying, ‘Oh, we’ll give this the deepest possible consideration.’”

Since then, there have been signs of moderation. Labor is committed to a referendum to place a First Nations voice in the constitution, although the signals from the government are unclear. Attorney-General Christian Porter says the voice to parliament won’t require constitutional amendment, yet funds have been allocated in the budget to refine a proposal that can be taken to a referendum. Prime Minister Scott Morrison kicked off the misinformation, claiming an Indigenous voice would be a third chamber and it was not on.

Kirby is keen to put that notion to bed: “This is complete misunderstanding that it would be a third chamber of the federal parliament – this is simply not true.” A voice to parliament should be uncontroversial – “the Australian Law Reform Commission is a voice to parliament, the auditor-general is a voice to parliament. ASIC is a voice to parliament…

“As the first people of land they should have a special opportunity into the House of Tongues to have a particular voice on matters that are relevant to their interests and their community,” he says.

The retired judge thinks one explanation for the confusion is that the people who drafted the Uluru statement were “speaking in semi-poetical language”. “It’s very vague and to somebody brought up in legal drafting it isn’t very specific.”

Possibly, he is being characteristically too kind, as there are likely also to be base reasons to generate confusion and discord for a proposal that needs to be treated with respect.

Religious freedom is another looming prospect with plenty of rough edges and much room for divisiveness. The mood in some sections of the Coalition was captured by National Party backbencher Barnaby Joyce, who declared that employment contracts should not be allowed to inhibit people such as Israel Folau publicly condemning people to hell on religious grounds.

Kirby rose to the occasion: “Well, I rather respect Barnaby Joyce’s views on country matters. He’s a bit like Senator [Bill] Heffernan, you know, he really does know things about farms and water and things like that, and city slickers have to listen to him on that. But, excuse me, on the matters of essential morality and sexual morality he’s not a person who I would naturally go to and listen to on those issues.”

If religious freedoms are to get a Pentecostal makeover by parliament, Kirby says legal protections for religious schools are a complicated prospect.

“When you start to do things that affect others then you have to reconcile competing human rights, including the competing human right especially of children and of minorities who are affected, including the LGBTQI minority and this then raises some issues of controversy,” he says.

“One issue of controversy is should it be possible for a religious school to teach that gay people are an abomination and that they should be put to death.”

We’re entering dark territory as he embarks on a dissertation about Genesis, Lot, pillars of salt and Sodom.

“If we look at our country, we don’t have a great glorious record in treating minorities. We’re pretty good in dealing with majorities because majorities cast the vote and determine the marginals and resolve the issue of who has power, governmental power.”

It does seem that we take our time to make progress with Indigenous Australia, race, people with disability, the aged and LGBTQIA minorities. Kirby has felt this acutely himself. Last year he told an audience at James Cook University in Townsville that “every time my name was raised in cabinet, Bob Hawke, then in his ascendancy as Australia’s prime minister, made it clear that he would not appoint a homosexual to the High Court”.

It was left to Paul Keating to make the nomination when Sir William Deane went from the court on Lake Burley Griffin to Government House at Yarralumla.

Like so many others, Kirby hid his homosexuality – “I went along with that for a long time”. It was his partner, Johan van Vloten, who decided enough was enough. “We have to get rid of this,” he said. “It is ridiculous, nothing will ever change while we go on conspiring in the ignorance of the straight majority by pretending we are straight.”

At the time Kirby argued it would be better not to reveal the truth until after he left the High Court. There is a great deal of “institutional feeling that you mustn’t do anything that would damage the court”.

Yet, it is precisely because he was a member of the court that ultimately he thought reality should be faced.

Kirby’s grandfather, who came to Sydney from Northern Ireland, took up the craft of journalism and worked on The Land newspaper and later The Sydney Morning Herald. “As a result of that I had running, coursing, though my veins the desire to communicate,” says Kirby.

Since leaving the court he’s been as vibrant and giving as ever, dedicated to spending as little time as possible out of the spotlight. There’s plenty to do in the judicial afterlife. “I’m not just sitting around pining for a case to do,” he says, “or drinking cups of tea and talking to journalists.”