New York Mayor Michael Bloomberg took a shot at those criticizing the construction of a Muslim community center and mosque near Ground Zero on Tuesday, asking rhetorically whether a new mosque could ever be built near the property that once held the World Trade Center.

“The question will then become how big should the no-mosque zone around the World Trade Center be?” Bloomberg told a Muslim-American audience at a dinner celebrating the breaking of the fast during Ramadan. “There is already a mosque four blocks away. Should it, too, be moved?”

While opponents argue that Bloomberg mischaracterized their arguments (no one is calling for the mosques already near Ground Zero to be shut down), it does raise the question of how far any new mosques would need to be from the site to avoid such a tremendous national uproar.

Many of the project’s prominent critics have said that Park51, the group heading up the site, is building the Muslim community center “too close” to Ground Zero. However, few of them have specified, (even when pressed), just how far away the mosque needs to be in order to receive their approval. Many of them oppose it on grounds that it is insensitive to the memory of those who lost their lives in the September 11, 2001 terrorist attacks. But when asked to suggest a specific distance that the site needs to be from Ground Zero, few are willing to elaborate further.

Illinois Democratic Gov. Pat Quinn, for instance, said last week that there should be a “zone of solemnity” around sites like Ground Zero, but did not specify how far such a zone should spread.

“I do believe that there are special places on Earth that should have a zone of solemnity around them,” he said last week. “I would strongly urge those who are thinking of putting a mosque within that zone to rethink their position.”

When asked for specifics, a spokesman from the Illinois governor’s office pointed to the August 20th press conference in which Quinn addressed the issue. While the governor did explain his reasoning during the briefing, he did not provide a precise distance. A request for a further explanation from the governor’s office went unanswered.

Former Republican House Speaker Newt Gingrich has said that he would approve of a mosque if it were near Central Park, which is just beyond four miles off the site. Likewise, former Republican Vice-Presidential candidate Sarah Palin has criticized the mosque’s position on grounds that it is “steps away from where radical Islamists killed 3000 people.”

Additionally, a spokesman for House Majority Leader Harry Reid, Nevada Democrat, simply said that Reid thinks the community center should “be built someplace else.”

So how many “steps away,” exactly, would a mosque need to be to avoid controversy?

It’s just not that simple, said Robert Spencer, author and editor of the website Jihad Watch, adding that it would be impossible to pin down an exact appropriate location for an Islamic center in the neighborhood.

The proximity to Ground Zero is just one component of a wide range of factors that ought to be considered, he said. These include “the historical connections of the new site to 9/11 and the buildings in the surrounding area.” Also, the site would need to be “as far away as would be necessary to take away the symbolic value that Imam Feisal Abdul Rauf and Daisy Kahn have alluded to in saying that this mosque is intended to make a statement about 9/11.” With that in mind, he said, any answer with just a specific distance that did not provide for the other elements would be incomplete.

“I’m not going to give you an address. There is no way I could possibly do that or anybody could do that,” he replied when asked during a phone interview. “…You’re trying to trap me and I know it. You want to play the game? I know how to play this game. I’ve been doing this for many years, alright? I’ve talked to lots of reporters, I know the games you play. I ain’t playing. You’re trying to get me to give you an address and say ‘oh, if it’s one block over or one building over then it’s okay with Spencer, but one building over here, no then it’s a triumphal mosque.’ Well I’m not playing.”

While that may explain why Park51 opponents aren’t talking specifics, some have expressed ideas about what distance they would deem appropriate.

Dr. Richard Land, president of the Southern Baptist Convention’s Ethics and Religious Liberty Commission and an opponent of the project, said that while he disagreed with the New York Landmarks Preservation Commission decision to approve the project, he did think there was a more proper distance.

“My guess is two or three blocks further away,” Land said. “Something that would not be within eyesight if it weren’t for intervening buildings or that you couldn’t hit with a rock from Ground Zero.”

Land added that he could not speak on behalf of his commission, but was simply commenting as a “regular citizen.”

Others say that distance from Ground Zero is not even the major issue at play. Instead, it is more a matter of the size and ambition of the project.

“I would not say that there is a specific distance that should be maintained between any given mosques and Ground Zero,” said Stephen Schwartz, executive director Center for Islamic Pluralism, a Washington-based organization that has published articles arguing against the mosque’s construction. “The problem with the Park51 project is its size and ambitious, if not overbearing character. Even had there been no controversy, the large building, its definition as an Islamic Center with a mosque accommodating 1,000 people at Friday prayer, and the ambitious publicity for it underscored the question of insensitivity.”

Schwartz continued: “If the Rauf-El Gamal plan goes forward, it should be located in a place where it will not call attention to the horrors of September 11 and not stir conflict over Muslim/non-Muslim relations. Such a location could still be in lower Manhattan — but it must be carried out without the associations with 9/11 that Rauf and El-Gamal, I think naively, have attached to it.”

As for the rest of the country, a recent CBS News poll found that 71 percent of Americans do not think that building a mosque so close to Ground Zero is appropriate, though 67 percent acknowledge that the group has a constitutional right to do so.

Email Chris Moody and follow him on Twitter