Indigenous leaders say Prime Minister Malcolm Turnbull's latest remarks on constitutional recognition are offensive.

Mr Turnbull said constitutional recognition of a Voice to Parliament would be "contrary to the principles of equality and citizenship".

The former co-chair of the Referendum Council, Pat Anderson, is deeply disappointed with the Prime Minister's choice of words.

"Well, you know, what the hell does that mean?"

She is worried that Mr Turnbull is trying to shut down discussion about Indigenous recognition.

"It's such a poor, crude level of conversation and discussion," Ms Anderson said.

Mr Turnbull is under pressure to revisit the idea of a referendum to establish an Indigenous body to advise parliament.

Ms Anderson said the advisory body, which was the recommendation of the Referendum Council and the landmark Uluru Statement from the Heart, would help inform policy to improve the lives of Aboriginal and Torres Strait Islander people.

She said it is essential if the problems facing Indigenous Australians are to be addressed.

"The number of people in incarceration, the levels of suicide — this is not some intellectual construct here," she said.

"This is who we are and we need to have a say or some kind of input into the decisions that affect us."

Criticism coming from several fronts

Others like Rod Little from the National Congress of Australia's First People are also confused.

Mr Little said Mr Turnbull was not staying to true to his own words.

"The Prime Minister has advocated … doing things with Aboriginal Torres Strait Islander peoples, not doing things to us."

Sorry, this video has expired Turnbull says Referendum Council's recommendation 'not a model that is desirable' (Image: AAP: Daniel Munoz)

The Prime Minister is facing a backlash from non-Indigenous groups, too.

Former Australian of the Year, Fiona Stanley and chief executive of the Australian Council of Social Services, Cassandra Goldie, have released a joint statement, calling on the Prime Minister to rethink his decision.

"We've got almost 1,000 individuals and organisations who have come together very quickly to show our strong support for the consensus that has been achieved through the Uluru statement," Ms Goldie told the ABC.

Assistant Health Minister Ken Wyatt, the first Aboriginal member elected to the House of Representatives, said people are frustrated with the lack of progress.

"I am receiving phone calls from Indigenous people from all over the country who are angry," Mr Wyatt said.

But he does not think the Prime Minister is walking away from the issue.

Mr Wyatt said he remained optimistic about the likelihood of a referendum on recognition, but the question is what form that referendum would take.

"The tit for tat and the toing and froing is not helpful," the Minister said.