GELSENKIRCHEN, Germany—Werner Gallmeister, a high-school headmaster here, called the police in January with a worrying message. A 16-year-old student known for his Islamist sympathies had been showing off a smartphone video of an explosive device.

Police already knew the teen. He had been interrogated and suspended from one school after threatening to “break the neck” of a Jewish student. Police in 2015 searched his home. And he was enrolled in a government-sponsored program designed to prevent radical youngsters turning to...