Aboriginal people have been dealt a "cold slap in the face" due to policy failures and the rejection of a referendum push, a former federal Aboriginal affairs minister says.

Key points: Former politician Fred Chaney says the Government's reaction to the Uluru Statement of the Heart was "a tragedy and a terrible mistake"

Former politician Fred Chaney says the Government's reaction to the Uluru Statement of the Heart was "a tragedy and a terrible mistake" Indigenous Affairs Minister Nigel Scullion has held "cabinet" talks with senior Yolngu leaders

Indigenous Affairs Minister Nigel Scullion has held "cabinet" talks with senior Yolngu leaders Duane Fraser, a Wulgurukaba and Bidjara man from Queensland, says Indigenous people are impatient for change

At the Garma festival in Arnhem Land, former Liberal politician Fred Chaney urged government departments to repair relations with Indigenous communities.

At Garma, Mr Chaney took bureaucrats to task over what he described as a series of policy decisions that had driven people further into poverty.

"There is real poverty and hardship and a loss of capacity in remote communities," he told the ABC.

"There are elements of the public service that have been very frank that this is a real area of difficulty for them."

Australian Institute of Aboriginal and Torres Strait Islander Studies chief executive Craig Ritchie said he understood many communities were fed up, but said departments were working to change.

"We can't keep doing things the same way," he said.

"It would certainly benefit future discussions having more Indigenous people involved in the public service particularly at senior levels."

'It's a wrong law'

Thousands of Australians are camped at the sacred Yolngu site, Gulkula, for the 20th anniversary of the Garma Festival this weekend.

In an emotional address at the festival's opening, Yolngu leader Galarrwuy Yunupingu said he was dismayed at hearing of Indigenous groups unable to access land and resources.

"It was a wrong law, and a wrong takeover. It must be given back to the landowners of every state," Dr Yunupingu said.

"The Government will be asked to do that.

"I don't think that's hard, it's going to be easy, if Aboriginal people have got their heads together."

At Garma, Indigenous Affairs Minister Nigel Scullion held "cabinet" talks with senior Yolngu leaders, which he described as a discussion between "nation-to-nation".

"It's a story of hope. If we can do it [here] then it is possible for every Aboriginal and Torres Strait Islander nation," he said.

Aboriginal dancers perform at Garma. ( ABC News: Owain Stia-James )

Disappointment Turnbull, Shorten didn't come

Over two decades, Garma has become Australia's key forum for debate on constitutional change.

A joint parliamentary committee is currently considering options to include Indigenous people in the constitution, and to give communities more power.

Hundreds of Indigenous people endorsed the Uluru Statement from the Heart last May — a call for a new elected representative body.

Fred Chaney urged government departments to repair relations with Indigenous communities. ( AAP: Paul Miller )

The Federal Government decided against holding a referendum to test the idea; a committee has begun a fresh round of consultations on the issue.

"What I think [Indigenous people] expected was that the Government would say: 'That's fantastic, let's sit down and talk about this,'" Mr Chaney said.

"Instead, they got a cold slap in the face, and I think that was a tragedy and a terrible mistake."

Duane Fraser, a Wulgurukaba and Bidjara man from Queensland, said Indigenous people were impatient for change.

He was disappointed the Prime Minister and the Opposition Leader were not attending Garma this year.

"It is unfortunate they haven't arrived. And the question is why? Why haven't they come?

"[It] definitely doesn't go unnoticed when our nation's leaders aren't present."