New York (CNN) -- With chants and banners, opponents of the planned construction of an Islamic community center and mosque near ground zero faced off Sunday with demonstrators in favor of the facility.

Hundreds of critics and supporters of the proposed center in New York showed up despite an overcast and drizzly sky to express their views amid the national debate over the facility.

Police estimated that supporters of the center numbered up to 250, and critics numbered about 450 during the demonstration.

iReport: See, share your images and opinions with CNN

A banner on the anti-center side said, "Land of the free. Stop sharia before it stops you," referring to Islamic law. Another sign read, "No mosque here. Preserve the dignity of our loved ones killed on 9/11."

Opponent Mike Meehan said he would not be opposed to the center if it were built elsewhere in the city, "just not down here, please. ... It's just too close to ground zero."

Another critic, Shawn Gilfeather, called it "a breeding ground for terrorists."

"I think the people that are backing, the people that are funding it, are actually in cahoots with them ... the terrorists," Gilfeather added. "I don't think they're just people practicing religion. I think there's something more."

Those in favor of the construction said freedom of religion was the main issue in the debate.

"There are many Muslims who lost Muslim family members at ground zero, so when they come to visit ground zero as a memorial, they should be able to walk two blocks down and pray for their loved ones," supporter Ali Akram said.

Others said Americans need to set an example of tolerance to the rest of the world.

"It would be giving in to bigotry and intolerance to demand that it be moved and I think in the end, it makes us less safe because we need to show the world that we are a tolerant, open society," supporter Ruth Massie said.

Lt. Col. Chris Dzubek, an Army reservist preparing to head to Iraq, agreed, saying developers have "the right and the ability to do it wherever they would like."

"That's what I like about the country is the ability for people who disagree with one another to hash it out without guns," Dzubek said.

At the conclusion of the protest, those opposed to the construction marched by ground zero.

The Islamic center's leaders say the $100 million facility calls for a community center including a mosque, performing arts center, gym, swimming pool and other public spaces.

It will be built near where the World Trade Center was destroyed by Islamic extremists on September 11, 2001. The attacks killed more than 2,700 people.

Some New Yorkers say an Islamic center near the site is a painful affront.

The Coalition to Honor Ground Zero organized the rally against the construction of the center. The NYC Coalition to Stop Islamophobia staged the counter-protest.

Plans to build the center near the site have stirred emotions nationwide. A CNN/Opinion Research Corp. poll released earlier this month marked nationwide opposition to the proposed facility at 68 percent.

"What that tells me is the wounds of 9/11 haven't healed, and I think if they haven't, perhaps we can find ways to bring about that healing by perhaps creating a situation where people will feel more comfortable," New York Gov. David Paterson said last week.

Paterson said there is no local, state or federal statute that prevents the construction of the facility.

On Sunday, Daisy Khan, who is heading the development of the project with her husband, said moving the project to another site is not under consideration for now.

She added that a move could be considered after consultations with "all major stakeholders."

"We have to be very careful and deliberate in making any move," Khan said.

CNN's Susan Candiotti and Ross Levitt contributed to this report.