Ken Wyatt says he fully expects some Coalition colleagues to cross the floor and campaign with the “no” case once the Morrison government presses ahead with recognising Indigenous Australians in the constitution.

But the minister for Indigenous Australians insisted Scott Morrison would stay the course on recognition despite deep splits within the government, that the prime minister would continue to pursue the change and not defer it, because putting this reform on the national agenda was Morrison’s idea.

Wyatt’s comments to Guardian Australia’s politics podcast came before Morrison used question time on Wednesday to play down the prospect of a referendum on recognition proceeding quickly. But rather than blaming the divisions within his own ranks for any delay, Morrison pointed the finger at Labor.

The prime minister said there was “a large gap between where the opposition stands on the form of this and where the government stands, and I would have thought there would need to be a real consensus to enable such a referendum to be successful”. Morrison said “timetables” weren’t important, but the ultimate success of the proposition was crucial.

Wyatt told the podcast that polling indicated a majority of voters strongly support recognising the first Australians in the constitution – even in conservative states such as Western Australia.

But he said the Canberra “bubble” created different dynamics, with political life isolating some parliamentarians from the sentiment prevailing outside.

“I normally don’t use [bubble] but I think there are times when the context in which we [parliamentarians] operate is a very unique environment,” he said.

“When you come into this place in the morning, you don’t see the rest of the world, when you leave at 9 or 10 or 2am what you see is the dark sky, the light you turn on before you flop into bed, and then you are up again.”

Constitutional conservatives, and some rightwingers within the Coalition, expressed strong opposition to the proposal immediately after Wyatt put recognition, and a voice to parliament, back on the political agenda last year.

There was more pushback in this week’s meeting of the Coalition party room, with three Liberal senators expressing negative views.

“I think the reality is, and I have to be brutally real, that there will be some colleagues who won’t support it,” Wyatt said. “They may run a very strong no campaign.

“I’ve got to respect there will be individuals who may take an absolutely rock-solid position that they will not budge, and that will be part of the debate, and they will have the right to mount a no argument.

“The barometer [of public opinion] is showing there is strong public support. But I need to work this through with everyone in parliament.

Asked whether he was confident Morrison would see it through given many of the current differences appear irreconcilable and the government is under pressure on a range of fronts, Wyatt insisted that pursuing this reform was the prime minister’s idea.

“The prime minister provided me with a charter letter. Constitutional recognition, the voice and truth telling are in that charter letter.

“That letter is the priority focus that I have to work on as a minister. If he didn’t have some degree of support for this, he wouldn’t have put it in my charter letter”.

Wyatt acknowledged that Morrison was concerned about the internal and external pushback. “The prime minister has been extremely supportive and he and I do talk and we are concerned that elements within society, including elements within our party, will fight us on this”.

“But we want to take a very pragmatic way forward.”

Wyatt insisted that he was being consultative, despite complaints inside the government that the minister was frontrunning both the cabinet and backbench committees by attempting to accelerate the timetable for a referendum.

Minister for Indigenous Australians Ken Wyatt (right) congratulates Scott Morrison after he presents the Closing the Gap report 2020 on Wednesday. Photograph: Mike Bowers/The Guardian

The latest round of blowback was prompted by comments from Wyatt that the government intended to hold a referendum by mid-2021. The minister told the party room this week those comments had been misrepresented.

Wyatt insisted he was talking to internal opponents. “We are engaging, we have been engaging for some time.”

While backing the right of colleagues to oppose the change if that was their considered view, he also urged critics to listen to the communities. He said constitutional recognition had a lot of parallels with the same-sex marriage debate inside the Coalition.

Wyatt said some colleagues expressed strong opposition to equal marriage before the plebiscite, but “what we saw is when the public had a say, we saw a quantum shift in support for same-sex marriage legislation”.

The comments came as Morrison argued on Wednesday that a new approach was needed to address Indigenous disadvantage as the latest Closing the Gap report found little progress has been made on five of seven targets, including life expectancy and child mortality rates.

Wyatt said the current approach had failed, and a more inclusive approach to measuring outcomes in Indigenous communities was needed in order to ensure progress was achieved across a range of indicators.

Morrison told parliament that despite the best of intentions, investments in new programs and bipartisan goodwill, Closing the Gap “has never really been a partnership with Indigenous people”.

“We perpetuated an ingrained way of thinking, passed down over two centuries and more, and it was the belief that we knew better than our Indigenous peoples. We don’t. We also thought we understood their problems better than they did. We don’t.

“They live them”.