A 'disappointed' Indigenous Affairs Minister Nigel Scullion says he won't make way for former prime minister Tony Abbott to take over his cabinet post, as prominent Indigenous community leaders also push back against the idea.

Mr Abbott confirmed through a surrogate on Tuesday that he wants Prime Minister Malcolm Turnbull to reshuffle the cabinet and appoint him Indigenous Affairs Minister - an area of longstanding interest to the former Liberal Party leader.

But to take up the post, Mr Abbott would need to oust the embattled Senator Scullion, who has come under fire for his clumsy handling of the Don Dale juvenile detention scandal and has at times struggled to retain the confidence of leaders in the Indigenous community.

"Senator Scullion has confirmed on a number of occasions he is absolutely committed to serving the full term as the Minister for Indigenous Affairs," a spokesman told Fairfax Media.