Civil rights groups have warned a vast, powerful system allowing the near real-time matching of citizens’ facial images risks a “profound chilling effect” on protest and dissent.

The technology – known in shorthand as “the capability” – collects and pools facial imagery from various state and federal government sources, including driver’s licences, passports and visas.

The biometric information can then rapidly – almost in real time – be compared with other sources, such as CCTV footage, to match identities.

The system, chiefly controlled by the federal Department of Home Affairs, is designed to give intelligence and security agencies a powerful tool to deter identity crime, and quickly identify terror and crime suspects.

But it has prompted serious concern among academics, human rights groups and privacy experts. The system sweeps up and processes citizens’ sensitive biometric information regardless of whether they have committed or are suspected of an offence.

Critics have warned of a “very substantial erosion of privacy”, function creep and the system’s potential use for mass general surveillance. There are also fears about the level of access given to private corporations and the legislation’s loose wording, which could allow it to be used for purposes other than related to terrorism or serious crime.

States agreed to the concept at a Council of Australian Governments meeting last year, though it is yet to be legislated by federal parliament.

New South Wales is one of the states in favour of the capability, and is legislating to allow state driver’s licences to be shared with the commonwealth and investing $52.6m over four years to facilitate its rollout.

A NSW parliamentary inquiry on Wednesday heard concerns that the system could have a chilling effect on political discussion, protest and civil dissent.

The NSW Council for Civil Liberties deputy president, Lesley Lynch, said the system’s ability for near real-time identity matching made it a “big stakes” transformation. She said it could allow mass general surveillance of the public, including during large gatherings.

“It’s hard to believe that it won’t lead to pressure, in the not too distant future, for this capability to be used in many contexts, and for many reasons,” Lynch said.

“This brings with it a real threat to anonymity. But the more concerning dimension is the attendant chilling effect on freedoms of political discussion, the right to protest and the right to dissent. We think these potential implications should be of concern to us all.”

The NSW government has previously denied the system would be used for mass public surveillance, and said its intention was for more targeted enforcement of identity crime.

Home affairs department assistant secretary, Andrew Rice, said identity crime was impacting on one in four Australians in their lifetime. The system, he said, was crucial in combatting such crime.

“Identity crime causes substantial harm to the economy and individuals each year and is a key enabler of terrorism and serious and organised crime,” he said.

The NSW’s privacy commissioner, Samantha Gavel, said the system had been designed with “robust” privacy safeguards. Gavel said it had been developed in consultation with state and federal privacy commissioners, and she expressed confidence in the protections limiting access by private corporations.

“I understand that entities will only have access to the system through participation agreements and that there are some significant restraints on private sector access to the system,” Gavel said.

Part of the system, the face verification service, is already operational. It allows a “one-to-one, image-based verification service” allowing one person’s photo to be matched against an image on one of their government records.

A second part to the system, known as the face identification service, will allow “a one-to-many, image-based identification service” that matches a photo of an anonymous person against multiple government records, to help establish their identity.

The database will be accessible to federal, state and territory governments through a central hub connecting the various photographic identity databases.