Burns, Oregon (CNN) The leaders of the Burns Paiute tribe have a message for the men and women who have taken over the Malheur National Wildlife Refuge outside Burns, Oregon: "Go home. We don't want you here."

The message came from several tribe members whose ancestors fought and died over portions of that land long before the ranchers and farmers had it, long before the federal government even existed.

The tribe is still fighting over land use but now works with the federal government's Bureau of Land Management to save its archaeological sites.

"We have good relations with the refuge. They protect our cultural rights there," said tribal council Chairwoman Charlotte Rodrique.

The Bureau of Land Management is the same agency that has riled up Nevada rancher Ammon Bundy and the armed protesters who joined him from out of state. The men took over the wildlife refuge headquarters, saying they would stay until the land was returned to who they consider its owners, the 100 or so ranchers and farmers who worked the land as far back as 1900.

"We are exercising our constitutional rights. We won't leave until these lands have been turned over to the their rightful owners," Bundy said. "More than 100 ranchers and farmers used to work this land, which was taken illegally by the federal government."

The Paiute tribe decided it was time to speak about what's happening at the refuge. They did so at length and with plenty of emotion.

"They just need to get the hell out of here," tribal council member Jarvis Kennedy told a crowd of reporters and local residents who showed up to listen to what the tribe had to say on the matter.

Later he explained why he felt so angry about the takeover.

"To me they are just a bunch of bullies and little criminals coming in here and trying to push us around over here and occupy our aboriginal territories out there where our ancestors are buried," Kennedy said.

He continued to tell the history of his tribe's fight over land. Members of the tribe are descendants of the Wadatika band of northern Paiutes. Their history in the area dates back 9,000 years ago, the tribe says. The ancestors of the Burns Paiutes lived in caves near the shores of lakes in the Northern Great Basin. When the lakes began drying up the tribe had to migrate.

The tribe said it has never ceded its right to the land but received federal recognition in 1868 and signed a treaty with the federal government that requires it to protect the safety of the natives and promised to prosecute any crime or injury perpetrated by any white man upon them.

Fast forward to 1879 after the treaty was signed. The Paiutes say their people were "loaded into wagons and ordered to walk under heavy guard" in knee-deep snow and forced off their land on foot.

"They literally walked our people, children and women off our lands. They had no problem killing us," Kennedy said.

Inside the Malheur National Wildlife Refuge headquarters, Kennedy said there are important official papers that document his tribe's history and his ancestors' existence on the land. That headquarters has been taken over by men from out of state.

"It gets tiring. It's the same battles that my ancestors had. And now it's just a bunch of different cavalry wearing a bunch of different coats," Kennedy said.

He stood with several members of the tribe, whose numbers in the Burns area have dwindled to about 200. They lead hard lives, many working odd jobs to try to make ends meet. Kennedy said he and many others rely heavily on federal government grants to survive.

"It's tough out here. Not a lot of jobs. If any company wants to relocate we'd welcome it," Kennedy said.

Bundy showed up with several others before Christmas to protest a legal case against local father and son Dwight and Steven Hammond.

A federal judge ordered the two men back to prison after a government appeal of their sentences for arson on federal lands. The government said the two set fires to cover up their poaching of deer.

Many of the townspeople said they are upset about what has happened to the Hammonds, seeing them returned to prison after serving the initial sentence, but there are mixed emotions about Bundy and the other protesters, who call themselves the Citizens for Constitutional Freedom.

A few ranchers have supported the group, bringing protesters warm soup and supper.

But in a community meeting called by the Harney County sheriff attended by hundreds of residents from the small town, the vast majority raised their hands when asked if they wanted the group to leave and the situation to end peacefully.

"What is going on down at the refuge is unfortunate. What led up to that is unfortunate, too. There are people here that tried to do a peaceful rally and that got highjacked." Harney County Sheriff David Ward told the crowd. "I am here today to ask those folks to go home and let us get back to our lives here in Harney County."

Louis Smith, who has lived in Burns for 56 years, said of the protesters, "They woke everybody up, we appreciate that. I don't agree with all they are doing, but I don't agree with the government's doing either."

After five days occupying the headquarters in the refuge that they broke into, including taking the tractors and using them as barricades at the entrance, the group has stayed put.

The tribal leaders said they agree with how the federal, local and state law enforcement authorities are dealing with the protesters. That has basically been waiting them out and not physically challenging them. The tribe members are quick to point out that they think things would be much different if it were them trying to take back the land the same way.

"We'd be already shot up, blown up or in jail. Just being honest; they are used to killing us," Kennedy said. "They are white men. That is the difference. That is just how I see it."