New York (CNN) -- Plans to build a mosque two blocks away from ground zero have set off an emotional debate among area residents and relatives of victims of the September 11, 2001 terror attacks.

Cordoba House project calls for a 15-story community center including a mosque, performance art center, gym, swimming pool and other public spaces.

The project is a collaboration between the American Society for Muslim Advancement and the Cordoba Initiative, both of which work to improve relations with followers of the religion.

The two groups presented their vision to part of the Community Board of lower Manhattan on Wednesday night.

Ro Sheffe, a board member who attended the meeting, said the project did not need to get the board's approval.

"They own the land, and their plans don't have any zoning changes," Sheffe said. "They came to us for our opinions and to let us know their plans. It was purely voluntary on their part."

The 12 members who were at the meeting voted unanimously to support the project. Community board members are appointed by the borough president and serve as advisers to the borough president and the mayor's office.

Daisy Khan, executive director of the Muslim society, described her vision of a center led by Muslims, but serving the community as a whole.

"It will have a real community feel, to celebrate the pluralism in the United States, as well as in the Islamic religion," Khan said. "It will also serve as a major platform for amplifying the silent voice of the majority of Muslims who have nothing to do with extremist ideologies. It will counter the extremist momentum."

The need for the center is twofold, Khan said, because it will support the needs of the growing Muslim community.

"The time for a center like this has come because Islam is an American religion," Khan said. "We need to take the 9/11 tragedy and turn it into something very positive."

Sheffe said a community center for lower Manhattan residents is "desperately needed." The area was mostly commercial, Sheffe said, but as more people move downtown, the lack of residential amenities is a problem.

The project got mixed reviews from families and friends of September 11 victims.

"I think it's the right thing to do," said Marvin Bethea, who was a paramedic at ground zero. "I lost 16 friends down there. But Muslims also got killed on 9/11. It would be a good sign of faith that we're not condemning all Muslims and that the Muslims who did this happened to be extremists. As a black man, I know what it's like to be discriminated against when you haven't done anything."

Herbert Ouida, whose son was killed in the attacks, supports the project as a way to bridge cultural divide.

"I understand the anger, the bitterness and hatred, but it only generates more hatred," Ouida said. "Such a large part of the world has this faith, and to say anyone who has this faith is a terrorist, it's terrible."

Others decried the idea of building a mosque so close to where their relatives died.

"Lower Manhattan should be made into a shrine for the people who died there," said Michael Valentin, a retired city detective who worked at ground zero. "It breaks my heart for the families who have to put up with this. I understand they're [building] it in a respectful way, but it just shouldn't be down there."

Others such as Barry Zelman said the site's location will be a painful reminder.

"[The 9/11 terrorists] did this in the name of Islam," Zelman said. "It's a sacred ground where these people died, where my brother was murdered, and to be in the shadows of that religion, it's just hypocritical and sacrilegious. "

However, Khan emphasized that the attacks killed Muslims, too.

"Three hundred of the victims were Muslim, that's 10 percent of the victims," she said. "We are Americans too. The 9/11 tragedy hurt everybody including the Muslim community. We are all in this together and together we have to fight against extremism and terrorism."

Cordoba House is still in its early stages of development. The American Society for Muslim Advancement is hoping to raise funds for the center to be completed in three to five years.