The Police Association of New South Wales has called for David Leyonhjelm to be sacked from a parliamentary committee into law enforcement after the Liberal Democrat senator criticised police for their treatment of football fans.

Leyonhjelm is heading up an inquiry into nanny state laws which he said restrict the freedom of adults to make their own decisions.

The inquiry on Tuesday looked at rules imposed on fans of Western Sydney Wanderers football club. It was revealed in September that police intended to ban fans from marching to the ground together, flying banners, clapping with their hands above their head and jumping up and down, among other measures. They later backed down from the proposals.

“There is a saying amongst them that all cops are bastards,” Leyonhjelm told reporters in Sydney on Tuesday. “The cops have earned that label, they have to un-earn it.”

The head of the Police Association, Scott Weber, labelled the comments “appalling” and “an outrage”.

“The senator, by repeating those comments and backing them up, is not acting how any elected official should behave. Such attitudes and vitriol are, and always will be, an affront to community decency and values,” Weber said.

Weber wants Leyonhjelm removed from the joint parliamentary committee into law enforcement.

“If senator David Leyonhjelm is representative of the thinking of a majority of senators then maybe it’s time to rethink if the people of Australia need a Senate at all,” he said.

Leyonhjelm was defiant. “The police are not our masters. They are our servants and I think they should remember that,” he said on Wednesday afternoon.

The justice minister, Michael Keenan, whose responsibilities include oversight of the Australian federal police, said the federal government does not have the power to sack parliamentarians.

“If we had that power we’d probably use it regularly,” he joked.

Keenan strongly backed law enforcement agencies, saying: “They’re not bullying us, they’re protecting us.”

He has called on all elected representatives to support them.

“I’m disappointed if it’s the case where individual members don’t do that, but really it’s a matter for those individual members,” he said.

Commercial radio commentator Ray Hadley took offence to Leyonhjelm’s comments, calling them “lunacy”.

Leyonhjelm was a “dimwit” and “captain of the dills in the Senate”, he said on his Sydney radio show on Wednesday morning. “Thinking and this bloke are foreign enemies.”

My advice to Ray Hadley and anyone concerned about being called a bastard, including coppers, is don’t be a bastard. — David Leyonhjelm (@DavidLeyonhjelm) November 4, 2015

On Tuesday, Leyonhjelm said that police should consider how other countries, including the United Kingdom, handle football-related violence.

“It’s not a problem with Wanderers fans,” he said. “It’s a problem with the way the police are handling it. If they [police] treat them all as criminals, as hoodlums, without any differentiation between them, they will naturally resent the police.

“They really do need to have a good hard look at other countries and how to do this better. They’re not handling this problem well.”

Leyonhjelm’s suggestion there should be a separate area in which fans let off flares was rubbished by Weber, who said it was dangerous.

A statement released by the Police Association of New South Wales said: “Anyone who thinks what has been happening at Western Sydney Wanderers games, and in the hours before them, is not dangerous needs to take a long hard look at themselves. Police are there to protect everyone – even these grubs.”

Leyonhjelm was making it more difficult to keep the community safe, Weber told Hadley on Wednesday morning.