Médecins Sans Frontières has been denied access to asylum seekers and refugees on Manus Island in order to assess their health and wellbeing, despite having been granted approval earlier in the week.

It comes as thousands of people attended snap rallies across Australia, calling for the men to be evacuated to Australia.

On Friday Papua New Guinea authorities forcibly removed the remaining refugees from the decommissioned detention centre after an almost four-week standoff during which hundreds of refugees refused to leave, citing fears for their safety in the Lorengau community.

Dr Stewart Condon, the president of MSF Australia and a member of the team sent to Manus Island, told Guardian Australia the humanitarian organisation had been trying to get access to the men since Wednesday, and called for authorities to let them in as an independent humanitarian observer.

“Every day since Wednesday we have been trying to get into the regional processing centre … or the transit centres and we’re not able to get past the gate,” said Condon.

Two doctors and a water and sanitation specialist arrived on Manus Island prior to the police operation, with approval from PNG immigration, but were denied entry to the centre.

“It’s quite clear – we had something in writing – that we were approved to get inside. We’ve been trying every day, and talking to as many people as we could to get that further approval. But we’re talking about possibly two governments, police, a private security firm, and we don’t have any read on why we haven’t been given any approval.”

On Friday authorities took the men to the East Lorengau transit centre, but about 60 were forced to sleep on the floor of a classroom because there wasn’t enough space for them.

Other alternative accommodation units remain construction sites, according to multiple independent agencies and observers. The Australian government has verbally disputed these claims.

Many asylum seekers and refugees in the decomissioned detention centre claimed they had been beaten by officers during the removal operation, including Kurdish Iranian refugee and journalist Behrouz Boochani who was specifically targeted for arrest on Thursday.

MSF was concerned about the health of the men who, after more than four years in detention, had eaten and drunk little for the last few weeks while living in unsanitary conditions. The men were also hit with bars by authorities during the removal operation, resulting in some minor injuries, and they alleged further beatings.

On Sunday MSF was able to see some of the men at the Lorengau hospital, according to refugees, but still not granted access inside the accommodation centres.

Condon said MSF had heard of some of the sick and injured men had been unable to access healthcare or were too afraid to leave their bed, and MSF called for access to visit them.

He said psychiatric concerns might be a “very significant” issue now given that prior to the standoff the men were believed to have been given a month’s worth of medication to see them through the transition.

Condon said he was concerned patients – particularly those with heart conditions and psychiatric illnesses – had now run out of medication.

MSF had also tried to access the health clinic set up for the refugees once they moved to Lorengau, but was again prevented.

“[Health clinic staff] came to the gate one day, but that’s the only meeting.”

He said he understood the clinic was only operational for limited hours during the day.

There have also been repeated claims of poor facilities and sanitation in the new accommodation units, which organisations such as MSF and the Australian Council for International Development have been prevented from accessing and verifying.

A group of refugees and asylum seekers walk by a construction site at the East Lorengau centre on Manus Island on Saturday. Photograph: Behrouz Boochani

MSF rarely speaks out publicly, but Condon said the situation called for it after the organisation had exhausted other options.

“We’ve come to the conclusion that we’re even more concerned about the medical state of these people, and it reached a point we were prepared to say we’ve done all the right things,” said Condon.

VIDEO: @drstewart_MSF reports directly from Papua New Guinea - "We can make an assessment and see the needs of these men and try to assist from a medical point of view." #Manus #ManusIsland #refugees #asylumseekers pic.twitter.com/lgo89MnOw3 — MSF Australia (@MSFAustralia) November 26, 2017

MSF has joined the UNHCR, the Australian Council for International Development, Amnesty International, Oxfam and other human rights groups in criticising the “humanitarian crisis” on Manus Island.

Despite pleas from the organisations and other governments including New Zealand, the Australian government has insisted the alternative accommodation is complete and refugees are simply trying to pressure it into bringing them to Australia.

The Turnbull government has rejected offers from New Zealand to resettle 150 refugees, saying it wanted to see out a previously struck deal with the US, and that New Zealand resettlement would help restart the people-smuggling trade.

On Sunday at the Bennelong byelection in Sydney, the prime minister, Malcolm Turnbull, attacked Labor’s candidate Kristina Keneally for once writing an article for Guardian Australia advocating Australia take the Manus Island refugees.

“[The people smugglers] will see and are seeing Bill Shorten’s choice of Kristina Keneally as Labor wanting to throw out the welcome mat for asylum seekers,” Turnbull said.

Protests were held around Australia over the weekend. Several thousand people gathered in cities including Melbourne, Sydney and Canberra on Sunday calling on the Australian government to evacuate the men from Manus Island.

Co-director of human rights at GetUp, Shen Narayanasamy, told the Melbourne rally international agencies and observers agreed it was a “humanitarian crisis”.

“This an emergency,” she said. “People’s lives literally hang in the balance. And what do our pollies say? ‘The US deal’. The US deal was a year ago and only 54 people have been settled.”

The Melbourne crowd later marched, with some blockading the Flinders Street intersection. Video filmed by one protester appeared to show police pushing people off the road.