Hundreds of thousands of people are at risk of being unfairly targeted by the police because their images are stored in a burgeoning facial recognition database, the UK’s biometrics watchdog has warned.

The commissioner, Paul Wiles, said facial images have gone far beyond their original purpose as custody records now that forces are using facial recognition software to police events.

South Wales Police courted controversy over summer by deploying the technology at the Champions League final in Cardiff. They matched the data against 500,000 local custody images.

The National Police Database now contains at least 19 million custody images, hundreds of thousands of which belong to people who were later acquitted or never charged with a crime, according to Wiles.

The retention of innocent people’s facial images was deemed unlawful by a court more than five years ago, but a Home Office review published in February said police should delete images only if the subject asks them to.

Wiles said it remained to be seen how many people would take up the opportunity. “The evidence from a similar application process to the police, to delete [Police National Computer] and biometric records, is not encouraging,” he wrote.

He told Radio 4’s Today programme this morning that it would be simpler if the police deleted images of innocent people who have been in custody when they are released, in line with rules applied to fingerprints and DNA.

However, Baroness Williams, a Home Office minister, said she thought the new regime struck “a reasonable balance” between privacy and public protection.

Wiles also warned that there is no independent oversight of the police’s retention of secondary biometric data such as facial images and voice recordings, as his remit covers only fingerprints and DNA.

“This whole process is in the hands of the police itself,” he said. “What we’ve got is a legislative deficit. I think that’s very worrying because if we’re not careful the public will lose confidence in the police.”

The role of the biometrics commissioner was established by the Protection of Freedoms Act 2010, after the European Court of Human Rights said retaining the DNA of individuals who are arrested but later acquitted or not charged is illegal.

The House of Commons’ science and technology select committee recommended in 2013 that the commissioner’s remit should be extended in general, but their wishes have so far been ignored.

A Home Office spokesperson said the government’s biometrics strategy did not have a publication date, but would be published “in due course”. It is nearly four years overdue.