A United Nations investigator probing discrimination against Native Americans has called on the US government to return some of the land stolen from Indian tribes as a step toward combatting continuing and systemic racial discrimination.

James Anaya, the UN special rapporteur on the rights of indigenous peoples, said no member of the US Congress would meet him as he investigated the part played by the government in the considerable difficulties faced by Indian tribes.

Anaya said that in nearly two weeks of visiting Indian reservations, indigenous communities in Alaska and Hawaii, and Native Americans now living in cities, he encountered people who suffered a history of dispossession of their lands and resources, the breakdown of their societies and "numerous instances of outright brutality, all grounded on racial discrimination".

"It's a racial discrimination that they feel is both systemic and also specific instances of ongoing discrimination that is felt at the individual level," he said.

Anaya said racism extended from the broad relationship between federal or state governments and tribes down to local issues such as education.

"For example, with the treatment of children in schools both by their peers and by teachers as well as the educational system itself; the way native Americans and indigenous peoples are reflected in the school curriculum and teaching," he said.

"And discrimination in the sense of the invisibility of Native Americans in the country overall that often is reflected in the popular media. The idea that is often projected through the mainstream media and among public figures that indigenous peoples are either gone or as a group are insignificant or that they're out to get benefits in terms of handouts, or their communities and cultures are reduced to casinos, which are just flatly wrong."

Close to a million people live on the US's 310 Native American reservations. Some tribes have done well from a boom in casinos on reservations but most have not.

Anaya visited an Oglala Sioux reservation where the per capita income is around $7,000 a year, less than one-sixth of the national average, and life expectancy is about 50 years.

The two Sioux reservations in South Dakota – Rosebud and Pine Ridge – have some of the country's poorest living conditions, including mass unemployment and the highest suicide rate in the western hemisphere with an epidemic of teenagers killing themselves.

"You can see they're in a somewhat precarious situation in terms of their basic existence and the stability of their communities given that precarious land tenure situation. It's not like they have large fisheries as a resource base to sustain them. In basic economic terms it's a very difficult situation. You have upwards of 70% unemployment on the reservation and all kinds of social ills accompanying that. Very tough conditions," he said.

Anaya said Rosebud is an example where returning land taken by the US government could improve a tribe's fortunes as well as contribute to a "process of reconciliation".

"At Rosebud, that's a situation where indigenous people have seen over time encroachment on to their land and they've lost vast territories and there have been clear instances of broken treaty promises. It's undisputed that the Black Hills was guaranteed them by treaty and that treaty was just outright violated by the United States in the 1900s. That has been recognised by the United States supreme court," he said.

Anaya said he would reserve detailed recommendations on a plan for land restoration until he presents his final report to the UN human rights council in September.

"I'm talking about restoring to indigenous peoples what obviously they're entitled to and they have a legitimate claim to in a way that is not devisive but restorative. That's the idea behind reconciliation," he said.

But any such proposal is likely to meet stiff resistance in Congress similar to that which has previously greeted calls for the US government to pay reparations for slavery to African-American communities.

Anaya said he had received "exemplary cooperation" from the Obama administration but he declined to speculate on why no members of Congress would meet him.

"I typically meet with members of the national legislature on my country visits and I don't know the reason," he said.

Last month, the US justice and interior departments announced a $1 billion settlement over nearly 56 million acres of Indian land held in trust by Washington but exploited by commercial interests for timber, farming, mining and other uses with little benefit to the tribes.

The attorney general, Eric Holder, said the settlement "fairly and honourably resolves historical grievances over the accounting and management of tribal trust funds, trust lands and other non-monetary trust resources that, for far too long, have been a source of conflict between Indian tribes and the United States."

But Anaya said that was only a step in the right direction.

"These are important steps but we're talking about mismanagement by the government of assets that were left to indigenous peoples," he said. "This money for the insults on top of the injury. It's not money for the initial problem itself, which is the taking of vast territories. This is very important and I think the administration should be commended for moving forward to settle these claims but there are these deeper issues that need to be addressed."