The outspoken government conservative Craig Kelly has warned he and other Liberal and National colleagues could “actively campaign for the no side” if the minister for Indigenous Australians, Ken Wyatt, pursues an ambitious proposal for constitutional recognition.

Kelly told Guardian Australia the government would be better placed working on practical matters, such as closing the gap between Indigenous and non-Indigenous Australians, and addressing high youth suicide rates, rather than contemplating “a separate body with people voting for people based on race”.

The Sydney Liberal said if Wyatt wanted to pursue “words in the constitution that don’t really mean anything, that are symbolic, then that’s fine”. But if the proposal included an idea like the “First Nations voice” set out in the Uluru Statement from the Heart then that was unacceptable to him, and he predicted it would be unacceptable to like-minded Coalition colleagues. “I think that idea is divisive.”

Ken Wyatt wants referendum on Indigenous constitutional recognition within three years Read more

Kelly’s intervention followed Wyatt using a major Naidoc week speech delivered at the National Press Club to put constitutional recognition back on the agenda after the election. Wyatt, the first Indigenous person to hold the portfolio, has promised to work towards holding a referendum on Indigenous recognition within three years.

The articulation of a timetable follows a meeting between Scott Morrison and Anthony Albanese last week in an effort to find an initial consensus. But while unveiling the three-year timetable, Wyatt made it clear he won’t stick to it if the referendum looks set to fail.

Wyatt used the press club speech on Wednesday to signal he wanted to persist with creating some model of the voice from the Uluru statement, but he indicated it might be legislated, rather than enshrined in the constitution, which was the position outlined in the Uluru process.

He said he wanted to co-design the voice model but, when pressed for detail, he indicated it could be a structure enmeshing existing state and federal representative bodies that would provide advice to the parliament. Anticipating internal pushback within the government, he counselled fellow Indigenous leaders to be “pragmatic” about what could be achieved.

While Labor has promised to try to land a bipartisan position, it is sticking by its recent election promise to pursue a voice enshrined in the Australian constitution.

“Labor has been saying clearly that we would like a voice entrenched in the Australian constitution, and I know that there are many people on the government benches that also believe that,” said Linda Burney, the shadow minister for Indigenous Australians. “And I think that is ultimately what we would want to see.”

Kelly said that would be a deal-breaker for him. “We’ve got good strong Indigenous representation at all levels of society, and I think you want people to think they are Australians rather than belonging to any racial group,” he said.

He said setting up separate structures, even if the representative model was legislated rather than enshrined in the constitution, risked creating “a reverse form of what South Africa was a few years ago”.