Islamic State has formed a number of clandestine cells in Libya a year after losing its main stronghold in the chaotic North African country, part of the militant group’s efforts to regroup on Europe’s doorstep.

The small cells, comprising up to several dozen fighters, have set up new bases outside Libyan towns in the past several months and started making money by hijacking commercial trucks and extorting migrant-smuggling rings, according to Libyan and European security officials.

Islamic State has also told fighters to go to Libya from Syria, where a U.S.-led coalition is pushing the terror group from its de facto capital of Raqqa, according to a defector and European security officials.

“They consider Libya to be the main entrance to Europe,” said Abu Baara al-Ansari, a Syrian who says he defected from Islamic State in June. Mr. al-Ansari said he worked in Raqqa for Islamic State in the office that tracked visitors to the group’s territory. He is now in Turkey and was interviewed via the Telegram messaging system.

The group’s efforts to stage a comeback in Libya after losing control of the coastal city of Sirte last year have sparked concern among European officials. Attackers who traveled from Syria to Europe have taken part in a number of deadly terrorist attacks in recent years, including in Paris and Brussels.