Federal senator Nova Peris has excoriated white Australia’s relationship with Aboriginal and Torres Strait Islander people, saying the government just gives “hope and aspiration on three-year funding cycles” and won’t allow Indigenous people to drive their own agenda.

Peris was speaking at the annual Garma festival on a panel of federal politicians, which included fellow Labor senator Warren Snowdon, who told the crowd he had argued heavily against Labor supporting the Coalition’s Northern Territory intervention.

The discussion, at the festival site in Gulkula, north-east Arnhem Land, followed a panel of remote community representatives who gave their own perspective on the impact of Indigenous policy and the intervention.

Peris said Aboriginal leaders and their communities were “screaming out saying: ‘we have the solutions’” but successive government policy did not let people drive their own future.

“The yardstick keeps moving for Aboriginal people,” she said. “We’re subjected to three year funding and when something works the rug gets ripped out from under our feet and we have to start all over again.

“You give hope and aspiration on three-year cycles.”

Peris said government language – such as the mantra to get kids to school, adults to work, and make communities safe – was patronising.

“Aboriginal affairs should be left alone. Don’t touch it for 10-15 years … Leave it alone and let Aboriginal people make decisions for themselves.”

The Coalition parliamentary secretary for Indigenous affairs, Alan Tudge – the only Coalition panelist after George Brandis withdrew – rejected earlier accusations there was no follow through on government rhetoric about improving Indigenous lives.

Aboriginal affairs should be left alone. Don’t touch it for 10-15 years … let people make decisions for themselves. Nova Peris

“We can always do more, and we do need to do more,” said Tudge. However “we’ve laid down some very clear markers, at least in relation to our core priorities, and we’ve done some very important structural change as well,” he said.

Tudge also pointed to government achievements in school attendance and education programs across the country.

Peris said Aboriginal people were constantly having to defend themselves because Australia would not accept the history of what was done to the first nations people. She pointed to the strong relationship between Australia and Japan despite a war between the two countries in the past.

“You can only move forward when you acknowledge the truth and right now this country has a problem with the truth of Aboriginal people,” Peris told the audience.

“My mother, a member of the stolen generation, my grandparents, members of the stolen generation, my grandfather, who had chains around his goddamn neck. That is the truth of this country but we keep closing the history books.”

“This government needs leadership and when you get people like the Andrew Bolts of the world who says the stolen generation is a myth – that’s bullshit. It happened and it’s real.”

Peris and Snowdon both criticised the Northern Territory intervention which was launched as emergency action – requiring the suspension of the racial discrimination act – by the Howard government in 2007 and continued by successive governments, including Labor.

Snowdon told the audience he “argued strongly” in the party room against supporting the legislation, but was defeated and so voted with the party.

Despite Labor’s initial support and its continuation of the policy when the party took office, “at its core, the intervention was wrong,” Snowdon said.

“I remember the very day that Labor took the decision – it was over lunchtime, I remember the discussion very vividly ... they said the government’s decided to do this, we think we need to support it,” he told the crowd.

“Internally I opposed it heavily. I argued and argued and argued against it. I’m a member of the Labor party, the decision was taken to support it, and therefore as a trooper and part of the team that’s what I did, but at the same time I was extremely critical of every element of it. And I remain critical of it.”

Panel member and the Labor spokesman for Indigenous affairs, Shayne Neumann, said the Labor government had made mistakes but “now we need to be looking forward”.

We call ourselves Australians and that’s how we should be treated – as Australians Yananymul Mununggurr

Prior to the political panel, audience members heard from representatives of Indigenous communities, who spoke of the feeling of disempowerment and alienation brought on them by the intervention.

Yananymul Mununggurr said the intervention was “a racist thing” for Yolngu people.

“It brought more anger, more sadness, more sorrow, because we Yolngu people felt our way of life, our control was taken away from us,” she said.

“The government just came in and took away our rights and left us with nothing. The intervention was another way of alienating Yolngu people to this nation. We were being treated as second class citizens and it was a disgrace for the government to do that to the first people of this nation. We call ourselves Australians and that’s how we should be treated – as Australians.”

The panel – on the topic of homelands – also heard from Bill Gray, an Indigenous affairs expert and former high-level public servant in the sector, who said the government and Indigenous communities both want the same thing when it comes to remote communities, education, and health, but there is a disconnect between government rhetoric and policy.

He said pressures on communities, such as poor infrastructure, top-down policies and regulations which prevented the building of more houses was sending a subtle message to Indigenous people that they are being encouraged to leave their homelands.