A team of European law-enforcement officials was hot on the trail of a potential terror plot in October, fearing an attack during Christmas season, when their keyhole into a suspect’s phone went dark.

WhatsApp, Facebook Inc.’s popular messaging tool, had just notified about 1,400 users—among them the suspected terrorist—that their phones had been hacked by an “advanced cyber actor.” An elite surveillance team was using spyware from NSO Group, an Israeli company, to track the suspect, according to a law-enforcement official overseeing the investigation.

A judge in the Western European country had authorized investigators to deploy all means available to get into the suspect’s phone, for which the team used its government’s existing contract with NSO. The country’s use of NSO’s spyware wasn’t known to Facebook. NSO licenses its spyware to government clients, who use it to hack targets.

On Oct. 29, Facebook filed suit against NSO—which has been enmeshed in controversy after governments used its technology to spy on dissidents—in federal court in California, seeking unspecified financial penalties over NSO’s alleged hacking of WhatsApp software. It also sought an injunction prohibiting NSO from accessing Facebook and WhatsApp’s computer systems.

NSO said it is vigorously defending itself against the lawsuit, without elaborating.