Harjit Jeji Shergill would not go so far as to say there was an inevitability to the murder of six Sikh worshipers at his local temple, but after years of people venomously calling him "Bin Laden" he feared it might come to this somewhere in America.

As Oak Creek's Sikh community grapples with Sunday's tragedy it is buffeted by shock and mourning for some its best-known figures. There is also appreciation for the police officers who stopped the gunman – including one who took several bullets – and almost certainly saved other lives.

But underpinning everything is a pouring forth of frustration, just short of anger, at what Sikhs in Oak Creek and other parts of the US say is the frequent assumption that because of their turbans and beards they are Muslims – with all the weight that carries since 9/11.

One Sikh leader said that is an assumption "with deadly consequences". Another said that the Oak Creek killings are the "collateral damage" of the al-Qaida attacks.

Jeji Shergill, 62, said that since 9/11 he has regularly been assumed to be Muslim and that routinely spills over into abuse. "They compare us to the Muslims and we're completely different," he said. "I own a gas station. I am working there. People, they call me Bin Laden. Then I explain to them: sorry, you are misunderstanding. You are mixing us up with the Muslims. You try to explain about the turban and the beard. They still call you Bin Laden."

Shergill showed no great sympathy or solidarity with American Muslims who themselves endure unjustified abuse in the aftermath of 9/11, saying that Sikhs were better off getting away from them. But his account of being derisively called Bin Laden was not uncommon in Oak Creek.

Satinder Singh, who lives close to the temple and left it shortly before the shooting because he had work to attend to, said he was subjected to verbal abuse on occasions but never felt physically threatened.

"Sometimes they call you a Muslim or Bin Laden. It's the way they say it. They are not fair to everybody," said Singh who immigrated from India 14 years ago. "But I have not seen violence before. We did not expect it. We are peaceful. We open our arms to everyone."

There are about 500,000 Sikhs in the United States but the community in Wisconsin is small, numbering about 2,500 to 3,000 families. Despite its relatively small presence in the US, the Sikh faith is the fifth-largest in the world, with more than 30 million followers. It includes a belief in one God and that the goal of life is to lead an exemplary existence.

Jastit Singh, of the Sikh American Legal Defense and Education Fund, said: "A lot of the challenges we face as a community are because people don't know who we are. They don't know what we stand for and they make assumptions. It's those assumptions that can have deadly consequences."

It is not clear whether the gunman, Wade Michael Page, targeted the Sikh Temple of Wisconsin in the mistaken belief that its worshipers were Muslim. He had ties to white supremacist groups and may have been motivated by a broader racial prejudice.

But the experiences of the Sikh community in Oak Creek, and incidents elsewhere in the US, means it will have to be convinced the shootings are not a legacy of 9/11.

A few days after the al-Qaida attacks more than a decade ago, a Sikh man, Balbir Singh Sodhi, was shot dead in Arizona by a man who assumed he was a Muslim. Other attacks followed and the passing of time has not diminished the false assumptions.

In April, 92 members of Congress signed a letter urging the FBI to collect information on hate crimes against Sikhs. The effort was led by a New York Congressman, Joe Crowley, who said the Sikh community "is acutely susceptible to violence because of their appearance".

Amardeep Singh, of the Sikh Coalition, said that in addressing audiences since 9/11 he usually poses a question. "When you see a turban and beard, what is the first thing you think of? The answer is inevitably what? I think of a terrorist. We've had 11 years where the turban is equated with terrorism," he said. "Is our community an intentional victim? … No, we're collateral damage."

Among those appealing for a change in attitude was Amardeep Kaleka, whose father, Satwant – a founder of the temple and its president – was shot dead on Sunday.

Keleka described how his father arrived in the US with little money and built up a flourishing petrol station business. He was, said his son, such an enthusiast for the "American Dream" that years ago he planted an enormous US flag in front of his house. "When we came home from high school that day we said: papa, this thing is an eyesore," said Kaleka.

But his father told them to look down the street and see the flags flying from other houses. "You're in America now," he said.

Kaleka said his father remained convinced that Americans would accept them as Sikhs. Many people in Oak Creek on Monday expressed sympathy and solidarity with the Sikh community as government offices and business lowered flags to half mast, and some churches, banks and individuals put up signs of support.

But for all that, on Monday Kaleka sounded sceptical about his father's optimism. "We should know that America needs to have cultural understanding for anybody it deals with. Ninety-nine percent of this nation is immigrants. We need to know each other," he said.