Indigenous Australians minister Ken Wyatt has pledged to find a consensus way forward on constitutional recognition and hopes to hold a referendum within three years to achieve it.

Wyatt, the first Indigenous person to hold the portfolio, said he will work with his opposition counterpart, Wiradjuri woman Linda Burney, parliamentary colleagues from all sides and the community to find “the right set of words” to present to Australia.

In his first speech to the National Press Club since taking over the portfolio later on Wednesday, Wyatt will expand on the plan to present a set of words for a referendum to the nation.

“We need consensus, but we need the right set of words,” Wyatt told ABC radio.

“What people don’t realise is whatever words you insert into the constitution, they can have significant implications way beyond the simple wording. High court judges would use the constitution as the basis for some of their decisions, many of their decisions, and the simple intent could be a complex challenge, or an issue for Australia.

“The Mabo decision was brilliant, but nobody ever thought the whole concept of terra nullis would have been overturned like it was. It is our rule book and it is an important rule book.”

But finding the right phrasing is not the only challenge facing Wyatt and Burney. Of the 44 referendums held in Australia since 1901, just eight have been successful.

Wyatt said having both the government and the opposition in accord gave any future referendum on Indigenous recognition a better chance. But he believes true success lies in finding the consensus “that our people agree to, that the majority of Australians will accept and the majority of states and territories”.

“It is a high hurdle to have to leap over but it is one that protects our democracy,” he said.

In May 2017, a gathering of 300 Aboriginal and Torres Strait Islander people signed the Uluru statement from the heart and presented it to the then prime minister Malcolm Turnbull and former opposition leader, Bill Shorten, proposing a constitutionally enshrined First Nations representative body to advise parliament on policy which affected Indigenous peoples.

It also called on a commitment to “truth telling” of Australia’s colonial history through the establishment of a Makarrata commission.

Last year Scott Morrison rejected the voice to parliament proposal, claiming it would constitute a third chamber, continuing a criticism Turnbull had used. Indigenous and legal experts have rejected that characterisation, pointing out a voice to parliament would not have any veto powers, nor would it examine every piece of legislation. The debate stalled any progress on Indigenous recognition.

Labor senator Pat Dodson, known by many as the “father of reconciliation”, told ABC radio it had been more than two years since the Uluru statement from the heart was delivered to parliamentary leaders and it was time to start moving.

“I think these are matters which have dragged on for far too long, and certainly the other aspect, which is the Markarrata commission of treaty making, is being lost in the course of the discussions here.

“People should really put the challenge to the minister, come up with the model that they want, the voice, and the set of words that they want to go into the constitution, and there has been proposals in relation to those matters and it shouldn’t take more than 18 months to settle that.”

Dodson also said he believed a treaty was crucial. Australia remains the only Commonwealth nation not to have signed a treaty with its Indigenous peoples.

“People have asked for certainty, they want the tyranny of their powerlessness, I think is part of the Uluru statement and they want to remove powerlessness from their interface with the government.

“Now just having a legislative entity doesn’t do that. You have to deal with the more substantial matters, the cause for the dispossession, and for the loss of culture and the loss of uniqueness of First Nations peoples. And the consequences of policies that have been perpetrated upon us over many years. That only comes through agreements.”

Wyatt said any future proposal would need to consider the Uluru statement from the heart.

“It has to, because, when I take the notion of the voice, that voice was not just a singular voice, it was voices from every level,” he said.

“I hear it frequently, even when I was in [the former portfolio of] aged care and I was out in Aboriginal communities talking about aged care. People were saying we need to have our voices heard.

“And it is a generic term that we have used, ever since I can remember in all of my activism since I was young, even through to now. I still hear people say we need a voice, so we can be heard.”

Wyatt said he was believed most Australians “want to right wrongs”, but said if he had colleagues who were opposed to the idea, he would respect their position on it. One Nation leader Pauline Hanson has already criticised the process, claiming that as a white person born in Australia, she is “Indigenous”.

“I had someone say to me ‘why should we make mention of a particular group in the constitution?’ What they overlook is that section 127 referred to natives not being counted. Section 5,126 talks about Indigenous Australians, so we have been in there right from the beginning, certainly in section 127. So we are not creating a precedent.

“The precedent was already set when the founding fathers wrote the constitution.”