New York (CNN) -- Protesters gathered in lower Manhattan mid-day Sunday to demonstrate against plans to build a mosque near the site of Ground Zero, where the twin towers of the World Trade Center were destroyed by Islamist hijackers on September 11, 2001.

Protest organizer Pamela Geller, a conservative blogger, and her group, "Stop the Islamicization of America," planned the event because, according to the group's website, "Building the Ground Zero mosque is not an issue of religious freedom, but of resisting an effort to insult the victims of 9/11 and to establish a beachhead for political Islam and Islamic supremacism in New York... Ground Zero is a war memorial, a burial ground. Respect it."

Geller said the NYPD and security at the rally told her about 5,000 demonstrators were there. But NYPD spokesman Sgt. Kevin Hayes said the police department's policy is to not provide crowd estimates and that he could not confirm Geller's number.

CNN iReporter Julio Ortiz-Teissonniere, who attended the rally and sent photos to CNN, said the number was closer to 200-300 while he was there for the first 45 minutes of the event. All three said the protest was peaceful.

Human rights advocates, politicians and families of 9/11 victims addressed the crowd. Both Geller and Ortiz-Teissonniere said those family members elicited a powerful, emotional response from the protesters.

Geller recently told CNN's Joy Behar that no one's telling the mosque's planners they can't build it, but "We're asking them not to."

"We feel it would be more appropriate maybe to build a center dedicated to expunging the Quranic texts of the violent ideology that inspired jihad, or perhaps a center to the victims of hundreds of millions of years of jihadi wars, land enslavements, cultural annihilations and mass slaughter," Geller said.

The project calls for a 13-story community center including a mosque, performing art center, gym, swimming pool and other public spaces.

It is a collaboration between the American Society for Muslim Advancement and the Cordoba Initiative.

The Cordoba Initiative aims to improve relations between Muslims and the West.

"The Cordoba Initiative hopes to build a $100 million, 13-story community center with Islamic, interfaith and secular programming, similar to the 92nd Street Y," its website says, referring to the cultural institution on the upper East side of Manhattan.

Daisy Khan of the American Society for Muslim Advancement told CNN it was a "community center with a prayer space inside."

She said the project was an opportunity for American Muslims living in New York to "give back" to the community.

"There is a lot of ignorance about who Muslims are. A center like this will be dedicated to removing that ignorance and it will also counter the extremists because moderate Muslims need a voice," she told CNN. "Their voices need to be amplified."

Local political leaders turned out in support of the community last month after Mark Williams of the conservative Tea Party Express reportedly said the mosque was for "the worship of the terrorists' monkey-god."

"To make room for peace there can be no room for hatred, bigotry or prejudice," City Councilman Robert Jackson said at the May 20 demonstration.

The project has the backing of the Community Board of lower Manhattan. It does not require city permission to go ahead.

iReport: Protests against mosque near Ground Zero

The plan has split people touched by the September 11 attacks.

"Lower Manhattan should be made into a shrine for the people who died there," said Michael Valentin, a retired city detective who worked at ground zero. "It breaks my heart for the families who have to put up with this. I understand they're [building] it in a respectful way, but it just shouldn't be down there."

Others such as Barry Zelman said the site's location will be a painful reminder.

"(The 9/11 terrorists) did this in the name of Islam," Zelman said. "It's a sacred ground where these people died, where my brother was murdered, and to be in the shadows of that religion, it's just hypocritical and sacrilegious. "

But Marvin Bethea, who was a paramedic at ground zero, said it was "the right thing to do."

"I lost 16 friends down there. But Muslims also got killed on 9/11. It would be a good sign of faith that we're not condemning all Muslims and that the Muslims who did this happened to be extremists," he said. "As a black man, I know what it's like to be discriminated against when you haven't done anything."

CNN's Nicole Bliman, Richard Greene and Mark Morgenstein contributed to this report.