(CNN) Police entered a decommissioned immigration detention center on Manus Island, Papua New Guinea Thursday and destroyed property and the dwindling food and water supplies of 400 or so refugees who are refusing to leave, according to some of the men there.

PNG and Australian authorities confirmed an operation was underway to try to convince the men to move out after a three-week standoff, with the refugees saying new facilities on the island don't give them enough protection from locals who don't want them living there.

About 40 police officers and 10 PNG immigration officials went to the center on Thursday, Dominic Kakas, Chief Superintendent of PNG Police told CNN. He described the asylum seekers as being "rowdy" but said officials were not using force against them. However, asylum seekers inside the facility allege police and immigration officials were being heavy handed.

The refugees who are still occupying the camp have had their belongings trashed, along with food and water stolen, by PNG police, they claim.

A video sent to CNN by a refugee from Pakistan named Naeem ud Din showed destroyed furniture littered throughout the detention center. He said officials were responsible for the destruction.

"They destroyed our food and what we have -- they destroyed our belongings," Ud Din, who says he received refugee status two years ago, told CNN in an audio message.

"Tonight we don't have any food, any water, nothing, because they destroyed everything. People are ready to eat leaves but people are not ready to go outside. It's a clear message to everyone... This is not humanity."

Damaged property in the Manus island refugee camp on Thursday November 23. Asylum seekers who remain in the shuttered camp claim that police caused the damage.

Australia: Facility trashed

Kakas, the chief superintendent denied the allegations of violence and said no property was destroyed or damaged. "What you hear is rubbish," he told CNN. "There are 50 of us and 400 of them, how can we use force?"

Kakas insisted officials were only talking to the asylum seekers to convince them to move out.

A total of 50 people were "peacefully relocated" to two alternative camps, a PNG police statement released later Thursday said. A Kurdish journalist Behrouz Boochani was "escorted out of the center by government officials." He was not arrested nor charged, Police Commissioner Gari Baki said in a statement.

"Everybody has been jumping up and down about the treatment of the refugees which is nothing but the best so far," Baki was quoted as saying.

"We are doing the best we can and the refugees cannot continue to be stubborn and defiant. The fact is that we are not moving them into the jungle. They are being relocated to two centers where there is water, electricity, food and medical services."

Immigration Minister Peter Dutton told Australian Radio Station 2GB Thursday confirmed a police operation was taking place and that he was keen for people to move out of the center, which was officially closed on October 31.

"I think it's outrageous that people are still there. They have trashed the facility, they are living in squalor and the Australian taxpayer has paid about 10 million dollars for a new facility and we want people to move," Dutton told 2BG Radio on Thursday.

He reiterated Australia's immigration and refugee policy, which denies entry to those who arrive by boat seeking asylum. "Under no circumstance will these people be coming to Australia and that's a final decision."

Heavy-handed, depressing and brutal'

The detention center was originally used to process people looking for asylum in Australia between 2001 and 2008. It was reopened in 2012 after a rise in the number of boat arrivals off Australia's shores. The refugees currently housed on Manus come from a range of countries, including Iran, Sri Lanka, Afghanistan and Myanmar.

A representative from a human rights organization who is on the island on a fact-finding mission told CNN by phone that the police action was "heavy-handed, depressing and brutal."

World Vision Australia's Chief Advocate Tim Costello said that he witnessed around 20 refugees being taken from the camp on buses Thursday morning, "looking demoralized." He says that the remaining men would continue to stand against enforced removal.

"They are very united, for the first time in four or five years they have agency, they are remarkably united and strong."

Costello laid the blame at the feet of the Australian administration.

"This is the Australian government's doing. We contracted out our problem to one of the world's poorest nations who didn't have capacity (to care for the refugees). That stain will remain very deep."

A tweet from the Australian Federal Police said that no member of its force was present during today's operations.

CLARIFICATION ON MANUS: The AFP has no member within the former Manus Regional Processing Centre and no involvement in today's actions. The AFP has a liaison officer in Manus Province working in an advisory and mentoring capacity to the RPNGC on policing matters. — AFP (@AusFedPolice) November 22, 2017

Damage

One asylum seeker at the facility, Walid Zazai, tweeting from the camp, alleged men were beaten.

"We can't take pictures or video. Whole area surrounded by them. When they see anyone filming they catch him, beat him and take him," he wrote, apparently referring to the PNG authorities.

We can't take pictures or video.

Whole area surrounded by them.

When they see anyone filming they catch him, beat him and take him. #SaveUS#ManusSOS — Walid Zazai (@ZazaiWalid) November 23, 2017

Refugees remaining in the center have been asked to move to two other locations -- the East Lorengau Refugee Transit Center or West Lorengau House -- both provided by the Australian government at a cost of up to $190 million (A$250 million dollars) a year.

Asylum seekers who haven't been found to be refugees are being asked to move to a third location, Hillside House.

The men say their safety is at risk at all three centers, which they claim don't offer the same level of security as the fenced detention center.

recent report from Human Rights Watch said the men had been "frequently assaulted" by "groups of local young men, often intoxicated and sometimes armed with sticks, rocks, knives, or screwdrivers."

Turnbull: 'We will not be pressured'

At a press conference Thursday, Australian Prime Minister Malcolm Turnbull urged those remaining at the shuttered facility to "leave and go to the alternative accommodation that has been provided," and added that the men should "obey the law and the lawful authorities of Papua New Guinea."

Refugees, photographed on November 23, who have remained at the Manus island refugee camp following its closure on October 31, 2017.

He added that the protest would not sway Australian authorities into changing immigration policy to admit the protesting men.

"We will not be pressured... our border security, the integrity of our borders is maintained by my government, it is maintained by my government and we will not outsource our migration policy to people smugglers."