Prime Minister Tony Abbott is facing a small but influential internal rebellion on the referendum to recognise Indigenous Australians in the Constitution.

Key points: Abbott facing internal dissent on referendum

Abbott facing internal dissent on referendum MP says failed vote 'could make it seem Australia is racist'

MP says failed vote 'could make it seem Australia is racist' Issue to come to head at WA Liberal conference

Issue to come to head at WA Liberal conference PM heads to Torres Strait on Sunday

A handful of Government MPs are completely opposed to any constitutional change, but a growing and broader group with the Coalition is questioning the merit of going ahead with the referendum as proposed in May 2017.

On Thursday the Prime Minister met four key Indigenous leaders - Noel Pearson, Patrick Dodson, Kirstie Parker and Megan Davis - to find common ground on the consultation process ahead of settling on a model and a question for the referendum.

After the meeting Mr Pearson said it was the "rumble we had to have to get the ground rules right, to get the understanding clear. But now we're very positive that we've got this show back on the rails."

When Mr Abbott survived a leadership spill earlier this year one backbencher privately said constitutional recognition was one of the "barnacles" the PM needed to remove.

With no agreement on a model or question to put to voters, the internal discontent has now grown.

Conservative senator Cory Bernardi has already voiced his view that an anti-discrimination clause would create a "lawyer's picnic" that would be "doomed to fail" at a referendum.

To succeed, the referendum needs to achieve support from the majority of states as well as the majority of voters across the country.

A number of MPs and senators are worried that threshold is too high and that the vote will be lost and deepen divisions between black and white Australia.

One said if the vote was lost it would "make it seem like Australia is racist".

Another backbencher said the referendum was headed into the "too hard basket" and it was a "no-win" situation for the Government.

Some of those who support the referendum are also downcast about whether there is enough public backing, even though a number of opinion polls have shown the majority of Australians support it.

Compounding this is a more general fear of voters being overloaded with the prospect of the upcoming federal election, a possible plebiscite on gay marriage and a referendum on constitutional recognition.

WA push for state conference to vote against constitutional recognition

The issue will again come to a head this weekend in Western Australia, ahead of the highly anticipated Canning by-election.

The Prime Minister will fly to Perth this weekend for the state Liberal conference, ahead of a week-long trip to the Torres Strait and Cape York region.

A motion by the party's policy committee has been put forward, proposing that: "The Liberal Party of Western Australia oppose any move to recognise a single race to the exclusion of all others in the body or preamble of the Commonwealth constitution".

Former WA Liberal leader Bill Hassell is among those who plan to support the motion if it comes before the conference.

"I don't think the Australian people will support it, however they dish it up to us," he said.

"However much of our money they spend trying to jam it down our throats."

Mr Hassell said the Prime Minister had misread the mood of the party.

The president of the Liberal Party's Cunderdin branch in Western Australia, John Snooke, is also opposed to constitutional recognition.

"I would really hate to see a situation where one race was elevated above another," he said.

"I have got no doubt it will inflame racial tensions and that would be a sad indictment on this country.

"My message to the Prime Minister is to not alter the constitution, it is an important document for this country, it should not be altered to suit politics of the day, it's a foundation document."

Mr Pearson said he also did not want a racial constitution.

"I will be in furious agreement with that resolution, in that I want to get rid of race as well," Mr Pearson told Lateline.

"What I do want recognition of is the fact that there were Indigenous peoples in this country prior to 1788.

"That is not a question of race, even though commentators like Andrew Bolt and so on try to conflate race and Indigenous. They are separate things.

"I completely agree that we shouldn't have [a] racial constitution. But the simple historical fact that there were peoples here that came upwards to 53,000 years ago ... that's a question of being Indigenous to Australia."

West Australian Liberal senator Chris Back said the question of whether to recognise Aboriginal people in the constitution was "very, very vexed."

He said he was concerned the process was not rushed and said he was very keen to hear the debate around the proposed policy motion at the state council meeting.

Senator Back said he felt there was currently pressure to "rush people to a position", which he feared would have negative consequences.