AKCAKALE, Turkey—Days after Islamic State vanished from the Syrian town of Tal Abyad, people of this nearby border settlement have been left with the lingering fear that retreating jihadists are now among them.

Officials and residents said they suspect fighters in disguise were among the thousands of refugees who crossed the border since Sunday, when Kurdish fighters first reached Tal Abyad. More than 23,000 refugees have come through Akcakale’s crossing this month.

Even before Islamic State lost Tal Abyad to Syrian Kurdish militia fighters and allied rebel groups, residents here said militants sympathetic to the group were known to use Akcakale as a way station to Syria. Akcakale is the last stop in Turkey on a key Islamic State supply line for fighters, weapons and other goods.

European intelligence officials say Hayat Boumeddiene—the wife of Amedy Coulibaly who was involved in the Paris terror attacks—crossed the border here, citing phone records traced to the area.

What has changed now is that many fear the town may be the target of revenge attacks for Islamic State’s stinging loss across the frontier, or provide a staging ground for attacks inside Syria. Few believe the fighting really ended with Islamic State’s rapid loss of Tal Abyad.