The Indigenous Advisory Council appointed by the prime minister to advise on “practical changes to improve the lives of Indigenous peoples” will meet Scott Morrison for the first time on Thursday, amid rising frustrations with the role of his special envoy, Tony Abbott.

The council is “really keen” to meet Morrison and get on with business, its co-chair Roy Ah See told Guardian Australia.

“We have a forward work plan, and a change in prime ministers really interrupts how we do our business,” he said. “It’s hard to build a relationship with leadership changes all the time. But we are hoping to be able to build a rapport with the PM.”

Abbott, Morrision’s special envoy on Indigenous affairs, will not be at the meeting but Ah See said he was welcome to meet with the council at a suitable time.

“That’s a matter for Mr Abbott, but the experts on Aboriginal affairs and on advice to the prime minister is the IAC,” he said. “We have a fantastic relationship with the Indigenous affairs minister, Nigel Scullion, and we’re looking forward to having some clarity around the role of special envoy at the meeting. Is the role going to help our communities?”

Ah See said there was a need to “future-proof” Aboriginal affairs by establishing a voice to parliament in the con­stitution “as a matter of priority” so that it could not be dismantled by the government of the day.

He also reiterated his views about Australia Day falling on 26 January, and Morrison’s suggestion for a separate day to celebrate Indigenous people.

“I’ve made no secret of my view that we have to change the date,” Ah See said. “The date has to change to celebrate 60,000 years, not just 250.”