Syrians and Iraqis have been fleeing their countries’ civil wars for years, but the refugee crisis grabbed international headlines last month when it forced itself on the European scene. Over 500,000 Syrian asylum seekers and thousands of Iraqis have gone to Europe as of September 2015, and that number is expected to climb dramatically. Some European governments reacted harshly, barring refugees from entering and restricting their transit, while others, notably Germany, are welcoming them with open arms – at least for now.

Debate about the refugees, both in Europe and the United States, often focuses on the question of terrorism: with the Islamic State raping and beheading with numbing regularity in Iraq and Syria, the fear is that admitting refugees from this part of the world will open the door to more terrorism and violence in Europe. Conservative voices in Europe have invoked the specter of Europe being flooded with “half a million” Islamic State fighters, while humanitarians dismiss predictions of a terrorism epidemic.

Both sides have it wrong. Concerns about terrorism and the refugees are legitimate, but the fears being voiced are usually exaggerated and the concerns raised often the wrong ones.

Terrorism and refugees share a long and painful history. Important terrorist groups like Fatah, Hamas, and the Popular Front for the Liberation of Palestinian (PFLP) grew out of the Palestinian refugee crisis of 1948. It took years, decades even, but eventually frustrated and politicized refugees, who had few opportunities to integrate into their host societies, joined militant groups promising a chance at liberation. The Afghan refugee community in Pakistan produced the Taliban. Somali refugees aided the terrorist group al-Shabaab. The list goes on.

Concerns about terrorism and the refugees are legitimate, but the fears being voiced are usually exaggerated and the concerns raised often the wrong ones.

Because the refugees are from Syria and Iraq, where the Islamic State is based, it is easy to conjure up fears that the jihadi group has inserted sleeper agents among the refugees who will burrow into host societies and then spring their trap. But the Islamic State doesn’t work that way. In its online magazine Dabiq and other propaganda organs, it stresses the ingathering of Muslims, though it does toss the occasional rhetorical bomb calling for Muslims already in the land of the infidels to “attack, kill, and terrorize the crusaders on their own streets and in their own homes.”

However, the Islamic State argues most “good Muslims” should travel to Iraq and Syria to fight on behalf of the Islamic State against its local enemies, not the other way around. (In contrast, Inspire, the English-language online magazine of al-Qaida in the Arabian Peninsula, stresses launching terrorist attacks in one’s home country.) The Islamic State might call for attacks in the West, but it has focused its own money, fighters, and suicide bombers on defeating its enemies in the Middle East. The refugees themselves, fleeing war and extremism, are not strong supporters of the most violent groups: if they were, they would have stayed in Iraq or Syria.

But Europe already has a terrorism problem, and the bigger danger is that radicalized European Muslims will transform the Syrian refugee community into a more violent one over time. Thousands of Europeans have gone to fight with the Islamic State in Iraq and Syria, and al-Qaida has long had a presence in Europe. These volunteers are sustained by radical preachers who condemn European ideals and support the idea of Muslims taking up arms. In addition, many European Muslims are alienated from their governments and societies, believing that as Muslims they never truly will, or should, belong.

The refugees themselves, fleeing war and extremism, are not strong supporters of the most violent groups: if they were, they would have stayed in Iraq or Syria.

If the refugees are treated as a short-term humanitarian problem rather than as a long-term integration challenge, then we are likely to see this problem worsen. Radicals will be among those who provide the religious, educational, and social support for the refugees – creating a problem where none existed. Indeed, the refugees need a comprehensive and long-term package that includes political rights, educational support, and economic assistance as well as immediate humanitarian aid, particularly if they are admitted in large numbers. If they cannot be integrated into local communities, then they risk perpetuating, or even exacerbating, the tensions between Muslim and non-Muslim communities in Europe. Despite their current gratitude for sanctuary in Europe, over time the refugees may be disenfranchised and become alienated. We’ve seen this movie before, where anger and disaffection fester, creating “suspect communities” that do not cooperate with law enforcement and security agencies and allow terrorists to recruit and operate with little interference.

The actual security risks now are low, but the potential ones are considerable if the refugee crisis is handled poorly. Policing, service provision, and local governance in general need to be provided for the long term. The worst thing European countries could do would be to invite in hundreds of thousands of refugees in a fit of sympathy and then lose interest or become hostile, starving them of support and vilifying them politically, creating a self-fulfilling prophecy.