Prominent human rights barrister Julian Burnside QC says he has been taken out of context by the Federal Government over his comments about trade union royal commissioner Dyson Heydon.

The Government has been quoting Mr Burnside in its defence of the integrity of Justice Heydon, who has come under fire for planning to give a speech at a Liberal Party fundraiser.

But Mr Burnside told AM that he maintained Justice Heydon should step aside and the Government was "grasping at straws".

In Question Time on Monday, Prime Minister Tony Abbott took to quoting Mr Burnside as he sought to defend the integrity of Justice Heydon.

"As was pointed out by someone who is no great friend of this Government, Julian Burnside QC, Dyson Heydon is a man of honour ... Dyson Heydon is a man of honour," Mr Abbott said.

Mr Burnside said he believed Justice Heydon was an honourable man but that the Government had quoted him out of context.

"They will grasp at any straws they can," he said.

"I don't think they would see me as a natural source of support, but the fact is I do have a high regard for Dyson Heydon.

"In fact, as I've said, I think he is an honourable person and I think in the circumstances an honourable person would step aside."

On Monday Justice Heydon, a former High Court judge, produced emails showing that he was originally told about the Liberal Party connection with the Sir Garfield Barwick address when invited last year.

But he said he overlooked it when he later confirmed that he would give this year's annual speech. He has pulled out of the address.

It has also been revealed that Justice Heydon was on the selection committee that awarded the Prime Minister a Rhodes Scholarship in the 1980s.

"The more his connections with the Liberal Party become apparent, the more difficult it is to give him the benefit of the doubt," Mr Burnside said.

"But I still think that he's an honourable person."

Possible to continue with a new commissioner: Burnside

The peak union group, the ACTU, is now considering an application for Justice Heydon's disqualification as royal commissioner.

ACTU spokesman Dave Oliver said unions believed Justice Heydon was biased, but they had not settled on their next step.

"We need to talk to many of our affiliates to get further legal advice, many of our affiliates have been involved in this royal commission and there's many teams of lawyers and we'll be sitting down over the short term to consider what all our options are," he said.

Mr Burnside said he believed it could be possible to continue with a new commissioner whose terms of reference could take into account the evidence already given.

He was counsel assisting the Australian Broadcasting Authority inquiry into the cash-for-comment scandal in 1999.

"The person chairing the inquiry, David Flint, was careless enough to go on air with John Laws just after the cross-examination of John Laws was not completed and he eventually saw the difficulty and stepped aside," Mr Burnside said.

"Within half an hour the commission had been reconstituted and it continued without a break using the evidence that had already been received.

"Now that's not the same as a royal commission, but it's an indication of what can be done."

The ACTU has until Thursday to make an application for his disqualification, with a hearing on Friday.