The federal government did not have time for an “endless public seminar” on setting up the royal commission into Northern Territory juvenile detention, the attorney general has said.

George Brandis made the comments on ABC’s AM program on Friday in response to claims the government had not consulted sufficiently on the terms of reference for the inquiry.

He also defended the participation of the Northern Territory government and the appointment of a former Northern Territory chief justice, Brian Ross Martin, to head the commission.

The royal commission, announced on Thursday, will consider the Don Dale facility and Northern Territory juvenile detention system but will not extend to other states and territories.

Labor’s leader in the Senate, Penny Wong, has criticised the lack of consultation and asked why an Indigenous commissioner was not appointed as well as Martin.

Brandis said he had consulted the Aboriginal and Torres Strait Islander social justice commissioner, Mick Gooda, and the prime minister, Malcolm Turnbull, had consulted his Indigenous advisory council chairman, Warren Mundine.

Brandis said his office had also been approached by Indigenous justice organisations.

Brandis said “we wanted to act swiftly and decisively”.

“And, frankly, I’m more interested in the royal commission getting to the bottom of the problem and coming up with actionable answers to the identified problems than having a debate about process,” he said.

The terms of reference were “comprehensive but focused”.

“The fact we didn’t hold some endless public seminar with any number of groups to talk about what should have been in the terms of reference is hardly the point,” he said.

Brandis defended consulting the Northern Territory’s chief minister, Adam Giles, saying Giles was “the first one to say there should be a royal commission”.

“This is a chief minister saying there should be a royal commission into his own government and former Labor governments,” he said.

Giles was cooperative and had, in fact, broadened the terms of reference.

Brandis said state governments often set up royal commissions to investigate abuses in their jurisdiction.

Asked about the appointment of Martin, Brandis said the NT government had not suggested him and “just because he may have sentenced juveniles does not mean he is not independent”.

“Any suggestion that he is not independent, not suitable for a task of this kind, is very foolish,” he said.

Speaking on Radio National on Friday, Wong said: “There appears to have been very little consultation with the community. That is regrettable. That is disappointing.

“Not only were the opposition not consulted, it appears many experts and advocates and communities in the Northern Territory have not been consulted.”

AM’s host, Ellen Fanning, said Gooda told Radio National on Thursday that he had contacted Brandis, not the other way around, and Brandis had called back, said he would “think about consultation” but did not call back again.

Wong said this showed Brandis had been “far too slippery”.

Wong said the “first order of priority is to deal with the territory”.

There should then be an opportunity to extend the scope of the inquiry if that was required because incarceration of Indigenous people was a national issue, she said.

The Australian Human Rights Commission president, Gillian Triggs, has recommended a two-phase inquiry to look first at the Northern Territory and then juvenile detention nationally.

Wong said the federal government should have considered appointing additional commissioners because the inquiry would “probably benefit” from an Indigenous commissioner.

At a press conference on Friday, the acting opposition leader, Tanya Plibersek, said the government should pause and “reconsider the rushed job they’ve done on the terms of reference”.

She said when she spoke to Turnbull on Thursday he had said “it was possible to amend the terms of reference during the course of the royal commission should that become necessary”.

“I am disappointed that that’s the attitude. I would have thought it would be good to get this right the first time.”

A Coalition frontbencher, Christopher Pyne, said it was crucial the royal commission be quick and smart.

“[It’s important] we don’t have a very long, drawn-out royal commission and by the time it actually hands down its recommendations everyone has forgotten what it was about at the beginning,” he said.

Brandis said the royal commission could consider and recommend criminal charges, despite the fact legal changes made in 2014 by the Giles government give guards immunity after the elapse of six months after an incident.