New York (CNN) -- Fareed Zakaria, the CNN host and Newsweek columnist, has returned a prestigious prize to the Anti-Defamation League, another rebuke for the esteemed civil rights group's opposition to an Islamic center near ground zero.

Zakaria, who received the ADL's Hubert Humphrey First Amendment Freedoms Prize five years ago, gave back the honor to ADL for publicly siding "with those urging the relocation" of the center, which would be located near the site of the September 11, 2001, attacks.

"I cannot in good conscience hold onto the award or the honorarium that came with it and am returning both," he said.

It is the latest in a wave of stinging criticism toward the Jewish organization's position from a range of people and groups, including those in the Jewish community, and the ADL said it was "saddened and stunned" by Zakaria's decision.

The ADL, which exists to fight discrimination, especially anti-Semitism, has said the center's backers had "every right to build at this site" and that "the bigotry some have expressed in attacking them is unfair, and wrong. But at the same time, it said "that building the center at the site will cause some victims more pain -- unnecessarily -- and that is not right."m

In a letter dated Friday to ADL National Director Abraham H. Foxman, Zakaria explained that he had been "delighted and moved" to be picked for the award "because of the high esteem" he had for the ADL.

"I have always been impressed by the fact that your mission is broad -- 'to put an end forever to unjust and unfair discrimination against and ridicule of any sect or body of citizens' -- and you have interpreted it broadly over the decades. You have fought discrimination against all religions, races, and creeds and have built a well-deserved reputation," Zakaria said.

That's why, he said, he was "stunned" that the ADL used its "immense prestige to take a side that is utterly opposed" to its own "animating purpose." He said the group's statements that it "must honor the feelings of victims even if irrational or bigoted, made matters worse."

Zakaria said he hopes his move will complement those who are urging the ADL "to reconsider and reverse" its stance.

"This decision will haunt the ADL for years if not decades to come. Whether or not the center is built, what is at stake here is the integrity of the ADL and its fidelity to its mission. Admitting an error is a small price to pay to regain your reputation," he said.

Foxman responded with a letter to Zakaria, saying he's "not only saddened but stunned and somewhat speechless by your decision."

"As someone I greatly respect for engaging in discussion and dialogue with an open mind, I would have expected you to reach out to me before coming to judgment," Foxman said.

Foxman reiterated that the group isn't opposing the right for an Islamic center or a mosque to be built and that it has and will continue to stand up for Muslims and others who are "targets of racism and bigotry."

"What we did was to make an appeal based solely on the issues of location and sensitivity. If the stated goal was to advance reconciliation and understanding, we believe taking into consideration the feelings of many victims and their families, of first responders and many New Yorkers, who are not bigots but still feel the pain of 9/11, would go a long way to achieving that reconciliation," Foxman said.

Foxman told Zakaria in his letter that he's holding onto his award and check "in hope that you will come to see that ADL acted appropriately and you will want to reclaim them."

The ADL this week said it won't fight the building of the center after New York City's Landmarks Preservation Commission eliminated a hurdle to the construction of the Islamic center two blocks north of the site of the attacks on the World Trade Center.

The commission denied landmark status for the building already at the site, moving the center one step closer to reality. The existing building is owned by the Cordoba Initiative, a Muslim outreach group, and already serves as a site where prayer services are held.

"We raised an issue ... but once the community board ruled, we move on," Foxman told CNN on Wednesday.

The Cordoba Initiative wants to demolish the existing structure and build a "$100 million, 13-story community center with Islamic, interfaith and secular programming, similar to the 92nd Street Y," its website says, referring to the cultural institution on the upper east side of Manhattan. The project calls for a mosque, a performing arts center, gym, swimming pool and other public spaces.

On Wednesday, Foxman, a Holocaust survivor, compared the ground zero controversy to the battle over building a convent near the Nazi concentration camp at Auschwitz.

"We're not challenging rights. We're asking, is this the right thing to do?" he said Wednesday.

CNN's Joe Sterling contributed to this report.