A team of European law-enforcement officials was hot on the trail of a potential terror plot in October, fearing an attack during Christmas season, when their keyhole into a suspect’s phone went dark.

WhatsApp, Facebook Inc.’s popular messaging tool, had just notified about 1,400 users—among them the suspected terrorist—that their phones had been hacked by an “advanced cyber actor.” An elite surveillance team was using spyware from NSO Group, an Israeli company, to track the suspect, according to a law-enforcement official...