Football fans are being recruited to join protests against Muslims. How worried should the authorities be?

The rise of the English Defence League, whose protests against Islamism have sparked violent city centre clashes, has been chaotic but rapid.

Three months ago, no one had heard of the EDL. But the organisation has risen to prominence in a spate of civil unrest in which far-right activists, football hooligans and known racists have fought running battles with Asian youths. The leadership insists they are not racist and just want to "peacefully protest against militant Islam".

Yet at EDL events, skinheads have raised Nazi salutes and other EDL supporters have chanted racist slogans such as "I hate Pakis more than you". One protest in Luton in May ended with scores of people attacking Asian businesses, smashing cars and threatening passersby.

Insiders have talked of plans to enlist football fans to march for the cause on the basis that "you need an army for a war".

With the organisation's confidence growing and plans for rallies in Leeds, Manchester and tomorrow in Trafalgar Square, concerned police chiefs and government ministers are asking what the English Defence League is, and what it wants.

It appears to have a hardcore of fewer than 200 in "divisions" in Bristol, Birmingham, Leeds and Luton. But those ranks are swelled by rightwing groups including gangs related to football clubs. Last night close to 500 had said they were considering attending the protest in London.

Its roots are modest, according to its self-proclaimed leader, a 28-year-old carpenter from Luton who goes by the pseudonym Tommy Robinson. He said the germ of the EDL was evident growing up in the Bedfordshire town.

"Everyone mixes until the age of 13 or 14 and then it stops and there are Asian dinner tables at school," he said. "I don't know what it is. Maybe their parents don't want them to mix."

Those separate tables are magnified in Luton today where a large part of the Muslim population lives in a network of streets around the main mosque in Bury Park. The town has had an unhappy connection with Islamist terrorism ever since four suicide bombers set off from there to attack London's transport system and kill 52 people on 7 July 2005.

Before that, there were tensions when a radical Muslim group protested in the town centre after the invasion of Afghanistan in 2001. Those feelings reached boiling point this March, when a small group of Muslim antiwar protesters held up placards at the homecoming of the 2nd Battalion Royal Anglian Regiment which read "Butchers of Basra" and "Anglian soldiers go to hell".

"The group that protested against the soldiers had been in the town centre since 2001," said Robinson. "In 2004 we held our own protest when we held a banner up saying 'Ban the Luton Taliban'.

"We were groups of friends and family, people who had gone to school with each other. We decided they could not be in the town centre again."

Only a handful of Muslim protesters disrupted the Anglians' homecoming parade, and they were drawn from an small extremist group that had already been ostracised by the mainstream Muslim community. However, it was enough for Robinson and others to set up a group called United People of Luton, and look across the country for support.

"We realised we didn't just want them off the streets of Luton, we wanted them off the streets of Britain," said Robinson.

Using Facebook, they forged links with a Birmingham-based group called British Citizens Against Muslim Extremists and quickly realised there was potential for a national organisation. "When we saw Birmingham's demonstration they were using the same slogans as us: 'We want our country back', 'Terrorists off the streets', 'Extremists out', 'Rule Britannia'. From there the EDL was set up."

Chief superintendent Mark Turner, of Bedfordshire police, said the group's aims were "really quite ill-defined".

That stems from the different interests that have rallied to a cause, which itself was named after the Welsh Defence League, set up by Jeff Marsh, a former football hooligan and convicted criminal.

The movement has been so fragmented that Robinson set up a website to drum up interest in March, while Chris Renton, listed as an "activist" on the BNP's leaked membership list, set up another EDL site. Paul Ray, another far-right activist from Dunstable, near Luton, broadcast video polemics on YouTube.

Ray's broadcasts on his Spirit of St George internet channel include his claim that a "very, very high proportion of the Muslim population is an Islamic extremist" and his description of Luton's Muslim community as "an al-Qaida enclave".

According to Robinson, none of these activities were co-ordinated.

By Ray's own account published on his Lionheart blog on the eve of the 8 August clashes in Birmingham: "The English Defence League that was originally built over many months and eventually set up by myself and others, was hijacked over the last couple of weeks leading up to tomorrow by a bunch of 'pirates' led by Chris Renton."

Around the same time, Trevor Kelway, a Portsmouth-based EDL supporter, became a spokesman for the organisation. In statements and phone interviews, Kelway pushed the line that the EDL was a peaceful, non-racist organisation, even promising that the last Birmingham protest would be "a great day out for all concerned".

When the day arrived EDL supporters were involved in running battles and police made 90 arrests.

Sharon Rowe, assistant chief constable of West Midlands, said the force had tried "everything they could" to liaise with the EDL before the demonstration but had been largely ignored. "If the EDL come back to this city I've got more of an evidence case and intelligence to therefore arrest them a lot earlier, to prevent a breach of the peace."

Fringe groups are rife in the world of the EDL, many of them have been established for far longer, and their beliefs often appear contradictory.

Davy Cooling, 26, a driver from Luton who helps run the EDL, admits to attending BNP events when he was younger, although he said he is not a member of the party.

"A few years ago I attended two or three meetings of the BNP in Luton, but I do not agree with their policy of banning black and other ethnic minoirity groups from membership. It doesn't matter what religion or race you are. Everyone is welcome to the EDL."

What does unite the group is a willingness to fight, said Robinson. "We feel that only people with that mentality will go [to demonstrate]," he said. "That's why it's all lads. Your upper class people won't stand there and get attacked, through fear.

"I am from the mentality that I am not going to back down. It started with what they did to the soldiers, but after that it has been about the two-sided treatment our community get compared to what the Muslim community get from the police and the council. The police hit us with batons and come at them with kid gloves."

This is not a version of events recognised by Bedfordshire police's Mark Turner: "We've had a series of marches where we have seen damage to property, we have seen people being assaulted, we have seen the odd racist attack – and that quite simply can't be tolerated."

Robinson said the group has recruited football supporters from clubs including Chelsea, QPR, Wolverhampton Wanderers, Aston Villa, Swansea and Cardiff.

But its lack of coherence has attracted the interest of those keen to harness the EDL's growing support.

They include Alan Lake, a London-based far-right activist who has advised Swedish nationalists on "countering jihad" and is advising the EDL in an attempt to broaden support with football fans and marshal events more carefully.

"We are catching a baby at the start of gestation," said Lake, who is considering funding the EDL. "We have a problem with numbers. We have an army of bloggers [in the far-right] but that's not going to get things done.

"Football fans are a potential source of support. They are a hoi polloi that gets off their backsides and travels to a city and they are available before and after matches." Observers from anti-fascist groups draw parallels between the EDL anger at Islamists and an earlier generation of football hooligans who supported loyalist paramilitary groups.

"In the 1980s and 1990s these hooligan groups perceived the threat to English masculinity was coming from the IRA's mainland bombing campaign," said Nick Lowles from Searchlight.

"They associated themselves with the loyalist movement and the chant 'no surrender to the IRA' was popular at football grounds around the country. Now many see what they term 'Islamist extremism' as the biggest threat."

Lowles said that while many of the hooligans involved were nationalists and racists, only a handful would associate themselves with fascist, far-right policies. "While it is not a fascist organisation, there are a handful of organised fascists in key positions We are concerned that as the EDL grows it will attract more extremists and fascists."

Lowles warned that the threat posed by the EDL should not be underestimated. "We saw the effects of hooligan incursions into Oldham and Bradford in 2001 and we must ensure that small groups of racists cannot whip up and incite this sort of trouble again.

"The authorities have a responsibility to local communities to protect them from violence. We have witnessed enough of the EDL to know that they want to whip up trouble wherever possible. They must nip this problem in the bud."

• This article was amended on Saturday 12 September 2009. An editing error in the print version resulted in remarks by Tommy Robinson being wrongly attributed to Davy Cooling. This has been corrected.