The French interior minister has said he should not have described May Day protesters “attacking” a Paris hospital as he faced calls to resign over the claim.

Critics have accused Christophe Castaner of spreading “fake news” after videos of the intrusion at the Pitié-Salpêtrière hospital appeared on social media and 32 demonstrators held by police were released without charge.

He on Friday acknowledged that “I should not have used the term” but he insisted that the intrusion, which happened at the end of the traditional union-led 1 May march through the French capital, was violent and remained under investigation. He described the row over his words as an “absurd polemic”.

Protesters insist they were seeking refuge from an advance by riot police and clouds of teargas, and were not targeting the hospital.

Castaner said he had spoken of an “attack” on the hospital after speaking to staff involved, but that the term “violent intrusion” used by the hospital director was better suited to the situation. “Use whatever term you prefer... ‘violent intrusion’ as used by the hospital director seems better adapted and appears to be confirmed by the videos we have since seen,” he told a press conference.

Footage shows demonstrators forcing a locked metal gate at the rear of the hospital at about 4.30pm. Around 50 protesters entered the hospital grounds and a group ran up a metal stairway and tried to get into the intensive care department.

Medical staff in the intensive care department blocked the door to stop them entering. A member of hospital staff inside the unit filmed the protesters, who were quickly removed by police.

Gwenaelle Bellocq, a nurse, said the drama was over in a matter of minutes. “It was very quick. We didn’t feel in any particular danger,” she said.

After the event, Prof Mathieu Raux, an anaesthetist at Pitié Salpêtrière, told journalists hospital staff were shocked by the intrusion, and that protesters had vandalised equipment in the surgical department. It was later reported that the damage had been done during an unrelated break-in the night before the march.

Staff said hospital officials had overreacted. “It wasn’t the nursing staff who were the most shocked but hospital chiefs who were concerned things would get out of hand,” a nurse told Le Monde.

Pitié-Salpêtrière’s director, Marie-Anne Ruder, had described the incident as “violent”.

Benoît Hamon, the former Socialist presidential candidate who heads the Génération.s movement that is fighting the European elections, accused Castaner of “manipulating information”.

“He only had to look at the videos to understand there was no attack, no violence, just people looking for shelter,” Hamon said. He admitted medical staff had “feared an intrusion in the intensive care unit”, but that it was wrong to suggest demonstrators had “attacked a hospital”.

“The government can’t fight fake news and tell lies at same time. If the government cannot produce proof of what it claims, I think the interior minister cannot remain interior minister,” Hamon said.