Western intelligence services have been warning that European and American jihadists have been flocking to Syria to fight. But they’ve been reluctant to say how many Americans have joined the extremist forces there—until now. The latest U.S. intelligence estimates say that more than 100 Americans have joined the jihad in Syria to fight alongside Sunni terrorists there.

Senior American intelligence officials tell The Daily Beast that they believe between six and 12 Americans who have gone to Syria to fight Assad have now returned to America. “We know where some are,” one senior U.S. intelligence official told The Daily Beast. “The concern is the scale of the problem we are dealing with.”

The scale of that problem by all accounts has gotten worse. Last fall, the official U.S. estimate on Americans specifically who have joined the jihad in Syria was in the low double digits. In January, the New York Times reported that at least 70 Americans have either traveled or attempted to travel to Syria. Earlier this month FBI Director James Comey told reporters that he believed “dozens” of Americans were suspected to be foreign fighters in Syria, but declined to give a more precise number.

In recent months, the U.S. intelligence community has made the tracking of all Westerners going to fight into Syria a top priority. Speaking in March before the Senate Foreign Relations Committee, Matthew Olsen, the director of the National Counter-Terrorism Center, described in vague terms an effort by the whole government to find Western citizens traveling to Syria and to track their travel.

“In light of the large foreign fighter component in Syria crisis, we are working together to gather every piece of information we can about the identity of these individuals,” he said at the time.

More recently, the issue of Western foreign fighters came up in top-level meetings between the Syrian opposition delegation and the Obama administration last week to Washington, D.C.

“We view all foreign fighters as a threat and they are not welcome. There is a convergence of interests between the moderate Syrian opposition and the international community in fighting these foreign fighters and insuring they do not use Syria as a launching pad for external attacks,” said Oubai Shabandar, a strategic communications adviser to the Syrian opposition’s foreign mission in Washington. “This was a major topic of conversation this month in meetings with the Syrian opposition delegation and top U.S. officials.”

The problem, U.S. counter-terrorism and intelligence officials tell The Daily Beast, is that there are just so many jihadists with Western passports traveling to fight in Syria that they worry some of them may slip back into the United States without being detected.

“The NSA does not have the ability to track thousands of bad guys—and on the human intelligence side, this is even more difficult,” another senior U.S. intelligence official told The Daily Beast. “So we are worried that people are slipping through the cracks.”

Olsen in his March testimony said there were thousands of foreign fighters in Syria and that hundreds of those fighters held Western passports.

“This raises our concern that radicalized individuals with extremist contacts and battlefield experience could return to their home countries to commit violence on their own initiative or participate in al Qaeda-directed plots aimed at Western targets outside of Syria,” he said. Olsen also said that a group of “al Qaeda veterans” from Afghanistan and Pakistan have gone to Syria, making the prospect of recruiting new members for the organization even more likely.

Aaron Zelin, a senior fellow at the Washington Institute for Near East Policy who closely tracks the flow of foreign fighters into Syria, said, “In the past when we’ve seen Americans go abroad to fight in foreign countries and a number of individuals have been trained to go back to attempt attacks on the homeland.” The best example he said is Faisal al-Shahzad, the Pakistani American who traveled to Taliban training camps in Pakistan and then attempted to set off a bomb in Times Square in 2010. Al-Shahzad failed to properly detonate his bomb and was reported to the New York police by a Muslim-American street vendor.

“It’s not just Americans who are going to Syria, but there are up to 3,000 European citizens from countries that have visa waivers with the United States who have also joined the jihad in Syria,” Zelin said. “This is why so many Western counter-terrorism officials are so worried, it’s much easier to get into our country with a Western passport.”

Those Americans that have gone off to fight in Syria also do not fit the typical terrorist profile. Last May, the Detroit Free Press reported that Nicole Lynn Mansfield, a convert to Islam, was killed in fighting in Syria fighting Assad. In April of 2013, a federal court charged Eric Harroun, a former U.S. Army private, with firing a rocket-propelled grenade while fighting alongside al-Nusra, al Qaeda’s official affiliate in Syria. If U.S. intelligence estimates are correct, these cases could be unfortunate harbingers of things to come.