Get breaking news alerts and special reports. The news and stories that matter, delivered weekday mornings.

LUTON, England — Stalls with colorful dresses and fresh vegetables spill out onto the streets of this suburb as children wearing traditional Islamic dress over their school uniforms run down pavements. Older generations greet neighbors or chat in groups, leaning against shop walls as they enjoy the early spring sun.

The quiet, unassuming town of Luton some 30 minutes from London by train is known mainly for its airport — one of the main international transport routes to the capital. But it has also developed a more dubious reputation as a breeding ground for jihadis.

Luton is home to 200,000 people, 50,000 of whom are Muslim — mainly of Pakistani, Indian and Bangladeshi heritage, according to 2011 census data. While only a tiny percentage of the population has felt the lure of radicalization, community leaders and experts point to several reason why extremism has crept into the area.

Two decades ago, notorious and now-jailed extremist preacher Anjem Choudary planted roots here, recruiting young men who often felt alienated by far-right activists. At the same time, out-of-touch religious leaders failed to draw them back to the moderate mainstream, some locals say.

Last week the British media reported that Khalid Masood, the 52-year-old attacker who killed an American tourist and three others outside the U.K. parliament, spent two years living in Luton.

Local media quoted a neighbor who said she remembered the attacker as an enthusiastic gardener who had lived opposite her between 2010 and 2011.

In the immediate aftermath of the attack, news Masood had links to the area was met with exasperation.

"When I heard about the attacks in Westminster I said I guarantee you this guy will have some link to Luton," said Amjad Iqbal, who added that he was frustrated that his town had once again been tied to extremism.

"He either parked his car here or he came here. Somehow it will have something to do with Luton," added the 42-year-old, laughing wearily.

Luton native Amjad Iqbal works at a money exchange store in Bury Park. Saphora Smith / for NBC News

The so-called 7/7 bombers, who killed 52 people on London’s transport routes in 2005, took the train to London from Luton on the day of the attacks.

In 2010 Swedish student Taimour Abdulwahab blew himself up in central Stockholm after attending university in the town. Three years later four men were jailed for plotting to send a remote-controlled car carrying a bomb into the town’s base for the Territorial Army — a British volunteer reserve force.

“Every far-right group that emerges wants to come and march in the town”

More recently there have been reports of entire families leaving the town to join ISIS in Syria, including one of 12 with a one-year-old baby. In 2015, Rahin Aziz, a key figure in Luton with a prominent social media presence, was killed in a drone attack in Syria after skipping bail, according to security sources.

In May, Junead Khan, who worked as a delivery driver and whose uncle, Rajib, was jailed for supporting ISIS, was jailed for life for plotting to run down and execute a U.S. serviceman. And in February, five men were sentenced to up to six years in prison after an undercover investigation found they had recruited for ISIS.

Muslim against Muslim

In Bury Park, a small area of Luton, around a dozen mosques stand in a warren of terraced houses, vegetable stalls and south Asian fashion stores.

Let our news meet your inbox. The news and stories that matters, delivered weekday mornings. This site is protected by recaptcha

Locals say the trouble started with Choudary, who started visiting Luton almost 20 years ago, recruiting members for the extremist Islamist group Al-Muhajiroun.

“Choudary was the spark,” said Iqbal, 42. “His guys would stand outside the bank on the corner and hand out leaflets."

A man walks down Dunstable Road in Luton, England. Tony Margiocchi / Barcroft Media via Getty Images

"We used to chase them off saying you're giving us a bad name," interjected his friend — a pharmacist with a star and crescent tattooed to his bald head, who declined to give his name.

Tanvir Munir, General Secretary of Luton Central Mosque said the leaflets promoted forming an Islamic state — a vision incompatible with the beliefs of most practicing Muslims.

Munir told NBC News that local religious leaders had complained to the authorities that Choudary's group was misleading and confusing their teens but claimed their concerns fell on deaf ears, as the organization's rhetoric was initially tolerated under the right to freedom of speech.

The owner of a local fabric store said the fault lay with the government.

"Why do we let people like Anjem Choudary into our country?" said the 53-year-old who identified himself as Ahmed M. "Blame the authorities if they're preaching bad things. Why didn't the authorities stamp them out?"

The trouble continued, Munir said. Fistfights started breaking out between local Muslims and Choudary’s men when the Luton Central Mosque had to begin “physically removing” the extremists from its vicinity, he said.

“It reached fever pitch during the Iraq war,” said Munir. “These people were English speaking, they were eloquent, middle class, educated to a university sort of standard, and were quite convincing in terms of their rhetoric."

The fact that Luton’s mosques mainly hired imams from the Indian subcontinent compounded this problem, Munir added. Many of these religious leaders did not speak English, and struggled to rebut Al-Muhajiroun’s political arguments and communicate with young Muslims who had been born and raised in Britain, he said.

“Both networks feed off each other and they are both feeding off the same human resource: young vulnerable people”

“We weren’t really getting to the youngsters,” Munir added.

What emerged, he continued, was a generation of young people going through an identity crisis with some becoming more western, going drinking and clubbing, and some becoming more insular and conservative.

Then came the far right.

The English Defence League (EDL), formed in 2010, started targeting Muslims in Luton in what former leader Stephen Lennon described as a “tidal wave of patriotism."

“Every far-right group that emerges wants to come and march in the town,” said Rehana Faisal, 39, who leads a Muslim women’s group in Luton. “They would invade our mosques, throw bibles at us, and stand outside with banners saying ‘stop the Islamization of Britain.’”

“It has an effect on young people to hear that hateful language, for it to be imported into your streets,” she said.

A woman sells traditional south Asian dresses in Bury Park, Luton. Saphora Smith / for NBC News

Faisal disputed the idea Luton has a problem with radicalization.

“There was a pocket of difficult individuals that we knew about and we worked with the authorities to sort the problem” she said, adding “some grew out of it, while a significant number have been arrested and been imprisoned.”

She said the perception of the problem is much greater than the reality on the ground.

“There’s an overwhelming feeling in this town that we really want this put behind us, that we want this town to move on,” Faisal added.

Luton officials declined to comment on the issue of radicalization to NBC News.

'Heads out of the sand'

Locals remain defensive about the attention their community has garnered and contend that the problems with radicalization are over. After all, many Muslims in the town celebrated when Choudary was jailed last year for encouraging support for ISIS.

Still, experts believe that the issue continues to dog the community.

"Some people might want to believe it's getting better, but unfortunately it's not. They're very much mistaken," said Hanif Qadir, an anti-radicalization activist and himself a former extremist.

“Anjem Choudary and his cronies have done a lot of damage and on top of that, to make matters worse, the EDL has been very effective in recruiting extreme right individuals,” he added.

For things to change in Luton, Qadir said Muslims need to get their “heads out of the sand,” and accept that there is a real problem stemming from within the Muslim community.

But also Qadir stressed the effort had to come from all parts of society and not just from Muslims. Divisive messages of the extreme right and extremists had to be neutralized.

He added: “Both networks feed off each other and they are both feeding off the same human resource: young vulnerable people.”