Gilles Bourdouleix, MP and mayor of the town of Cholet, faces expulsion from his party and legal action by the state

A centrist MP is facing expulsion from his party and legal action by the state after he allegedly said of French Travellers: "Hitler maybe didn't kill enough of them."

Gilles Bourdouleix, MP and mayor of the town of Cholet in the Maine and Loire region of western France, was reported to have made the comments on Sunday during an encounter with a group of Travellers who had parked over 100 caravans in a field owned by the local authority.

According to the local paper Le Courrier de L'Ouest, the meeting, in which Bourdouleix told the Travellers they would have to move on, was tense, with some Travellers accusing him of racism and a few making a Nazi salute towards him.

During the exchange, as reported by the newspaper, a man said to Bourdouleix: "You know how to talk better than [the former president Nicolas] Sarkozy." The MP replied that "the law must be implemented".

There was murmuring and Bourdouleix reportedly said in a hushed tone: "Just goes to show that Hitler maybe didn't kill enough of them."

When the MP denied the comments, the paper published a recording on its website. The 53-year-old insisted he had been the victim of a "scandalous montage" and was considering suing.

He told BFM TV: "I mumbled something like, 'If it was Hitler he would have killed them here,' meaning: 'Thank goodness I'm not Hitler and so there's no reason to call me Hitler.' This is shameful score-settling which aims to smear me." The newspaper said it absolutely stood by its reporting and the published recording.

The local police prefect's office has filed a legal complaint with a state prosecutor against Bourdouleix for glorification of crimes against humanity. At least 200,000 Gypsies are estimated to have been killed during the Holocaust.

The Socialist interior minister Manuel Valls said he hoped the MP would be sanctioned "very heavily". He said: "A case has been brought before the courts because this is praise for the crimes of the second world war, it's praise for Nazis, and coming from a mayor it's intolerable."

Praising crimes against humanity in France carries a maximum sentence of a year in jail and €45,000 (£37,700) in fines.

Bourdouleix belongs to the centre-right grouping the Union of Democrats and Independents (UDI), headed by the former environment minister Jean-Louis Borloo. The party will meet to discuss his possible exclusion on Wednesday.

France has one of the biggest Traveller communities in mainland Europe. There are around 350,000 French Travellers, a nomadic population which is different from the Roma community from eastern Europe.

Politicians from right and left united to condemn Bourdouleix's alleged comments. Valérie Trierweiler, the partner of the president François Hollande, tweeted that the comments were "intolerable" pointing out that one of the Nazi internment camps for Gypsies during the Holocaust was located in Montreuil-Bellay – only 60km (37 miles) from Cholet.

The prime minister, Jean-Marc Ayrault, said Bourdouleix's comments were "unacceptable, worrying" and "punishable by the law". Valérie Pécresse, of the traditional right-wing UMP party, said the comments were "utterly reprehensible" and parliament should sanction him.

Rights groups have warned of rising tension in France over non-settled communities, after several rightwing politicians recently spoke out against illegal caravan camps, using strong language that the left said was provocative.

This month Christian Estrosi, Nice's right-wing mayor, expelled one group of Travellers from a sports field, vowing to "crush" the "delinquents", and urged other mayors to revolt against what he called leniency by the Socialists.

Last month, rights groups said they would take legal action against the far-right politician Jean-Marie Le Pen after he told a meeting in Nice that Roma in the city were "rash-inducing" and "smelly".