A referendum to give Indigenous people more power to run their own affairs is a "life and death matter", Indigenous leaders say.

At the Garma Festival in north-east Arnhem Land, Gumatj clan leaders Dr Galarrwuy Yunupingu and Djawa Yunupingu said they were bitterly disappointed with the Government's treatment of the Uluru Statement from the Heart.

The statement — delivered by hundreds of people last May — called for a referendum to establish an Indigenous advisory body in the constitution.

Prime Minister Malcolm Turnbull last year rejected their proposal, saying it was short on detail and would not succeed at a referendum.

Yothu Yindi Foundation deputy chairman Djawa Yunupingu said the decision had been deeply frustrating.

Indigenous leader Galarrwuy Yunupingu at last year's Garma festival. ( ABC News: Mitchell Woolnough )

"How long do we have to wait to get this right? Another committee? Another hearing? Another meeting? Another Prime Minister?" he said.

He said his brother, the land rights leader Dr Galarrwuy Yunupingu, cried when he heard the Uluru Statement, which also called for a truth and justice commission and a treaty.

"So can we blame him for crying when the Uluru Statement is read to him and he knows it has been rejected?" Djawa Yunupingu said.

"The truth is that this sovereignty was not enjoyed by my people and it was put upon us by force and by fear of the superior power of the British people."

Cape York leader Noel Pearson said Mr Turnbull's rejection of the request to create a permanent Indigenous advisory body had "destroyed" people.

"We can't take the word of an ordinary person, we can't be intimidated, can't be told that the Australian people are so racist or redneck or opposed [to a referendum]," Mr Pearson said.

"This is life and death … this is no muck about on Twitter."

A joint parliamentary committee led by Labor senator Patrick Dodson and Coalition MP Julian Leeser is investigating options for reform.

This week its interim report said there was broad support for a "voice to parliament", with Aboriginal communities adamant that it should be elected with equal gender representation.

Mr Pearson said that the committee was doing important work and there was "some real hope that we might get to the destination we seek".

He urged the Government to reconsider its opposition to a referendum — otherwise an advisory body could be scrapped like the former Aboriginal and Torres Strait Islander Commission.

"It's got to be pinned down. Without it, we'll get pushed around," he said.

The 20th anniversary of the Garma Festival has kicked off in Arnhem Land. ( ABC News: Owain Stia-James )

But Indigenous Affairs Minister Nigel Scullion said his Government had "championed" constitutional recognition.

He said the idea for a Voice to Parliament needed more detail and clarity.

"What does that mean? Is it going to be a body that has a say on every piece of legislation?" Mr Scullion said.

"Will it have any say over policy and service delivery?

"Knowing that the Australian people are going to need a great deal of detail if they're going to support such a notion."

Thousands of Australians are at the Garma Festival this weekend, a 20-year-old forum for cultural education, debate and dance.

The theme this year is truth-telling. Australia's history was raised multiple times when the Referendum Council met with Indigenous communities across Australia last year.

Dr Yunupingu, who has negotiated with eight prime ministers over decades as a clan leader, said the first governor of New South Wales Arthur Phillip "didn't pass anything to Aborigines".

"Can you see anything in the law … where the Aborigines stand?

"They accounted for the kangaroos, the wallabies, the trees, the flowers, the birds.

"He merely rejected [us] … it didn't matter. There was some people, some dark people."