BRUSSELS—Two botched attacks in Europe in recent days signal that Islamist terror has entered a new phase, security officials say, one that is more disorganized and less sophisticated but risks spawning a growing number of assailants keen to kill with any means at hand.

When an Islamic State sympathizer tried to set off a bomb Tuesday night at a train station here, it failed to detonate with the intended force. Investigators suspect he used a faulty explosives recipe found online. The intended victims were able to flee, and...