David Becker / Getty Images

One of the issues with facial recognition today is its high error rates with people of color and women. But lawmakers on Wednesday pointed to a broader concern as well: Facial recognition's effects on civil liberties and privacy.

At the House Oversight Committee's third hearing on facial recognition, members of Congress looked to address the technology's use in public spaces, by both private companies and government agencies.

First, there's the well-documented tendency toward race and gender bias. Charles Romine, director of the Information Technology Laboratory at the National Institute of Standards and Technology, testified about a report from the agency last December that found a majority of facial recognition algorithms had a higher rate of false positives for people of color by "a factor of 10 to 100 times."

But several lawmakers on the committee were more concerned about how the technology would be deployed once it's perfected.

"If we only focus on the fact that they're not getting it right with facial recognition, we missed the whole argument, because technology is moving at warp speeds," said Rep. Mark Meadows, a Republican from North Carolina. "We're talking about scanning everybody's facial features, and even if they got it 100 percent right, how should that be used?"

The technology has been widely adopted by law enforcement, often cited in cases where it's helped find missing children or identify mass shooters. But police have also used it to monitor protesters, which both lawmakers and civil rights advocates argue has a chilling effect on First Amendment rights.

For example, the surveillance technology could be used to identify people at protests, and even the threat of that could intimidate citizens who want to exercise their right to free speech.

"While accuracy problems with face recognition are immense, it's absolutely true that even if accuracy increases, face recognition will still pose huge threats to civil rights and civil liberties," said Jake Laperruque, a senior counsel at the Constitution Project. "Absent rules, face recognition could be used to catalog people attending religious ceremonies, political rallies or protests."

Now playing: Watch this: How San Francisco's ban could impact facial recognition...

At the hearing, multiple lawmakers discussed the committee's plans to propose bills addressing facial recognition. While members of Congress have introduced legislation on the technology in the past, there aren't yet any federal laws on facial recognition in the US.

Both lawmakers and advocates anticipate that facial recognition will improve, and they pointed out that any legislation should focus on the future of the technology.

"When overseeing this technology, members of Congress must think about the broader civil liberties implications, not just whether or not the technology 'works,'" said Matthew Guariglia, an Electronic Frontier Foundation policy analyst.

While facial recognition has problems, even with complete accuracy, its effects are magnified with its gender and racial bias, Meredith Whittaker, co-director of New York University's AI Now Institute, said during the hearing. She discussed a case in Florida in which prosecutors used facial recognition as evidence to suggest a sentence of up to eight years in prison, despite the defendant never being able to challenge the technology's accuracy.

"There's no disclaimer to warn us that the populations already facing societal discrimination bear the brunt of facial recognition's failures," Whittaker said.

While a majority of the committee agreed that facial recognition's accuracy rates needs to be fixed, there was still the underlying concern about the technology's effects as a surveillance tool.

Instead of focusing on its racial and gender bias, lawmakers argued that facial recognition legislation should focus on protecting people's rights and privacy concerns.

"We're going to have to really grapple with what are the parameters of protecting privacy and controlling the use of this technology?" said Rep. Gerry Connolly, a Democrat from Virginia. "Irrespective of its accuracy, there are intrinsic concerns with this technology and its use."