By Daniel Burke, CNN Belief Blog Co-editor

(CNN) –– Declaring himself "American's most famous Catholic," comedian Stephen Colbert roasted church leaders at a charity event in New York on Thursday, taking aim at Pope Francis and Cardinal Timothy Dolan.

"As an observant Catholic, I believe the Pope is infallible," said Colbert, a Communion-class teacher at a parish in New Jersey. "But he's also wrong about a lot of things."

Colbert, whose bombastic persona on the "Colbert Report" often takes a conservative slant on Christianity, poked fun at the new Pope's humble lifestyle, saying that if the pontiff were in charge of the white-tie charity event, it would have been held at an IHOP, not New York's glitzy Waldorf-Astoria hotel.

“His Humbleness would be out washing the feet of the coat-check guy or something,” Colbert quipped. "We get it, you're modest."

"But it's not just his humble lifestyle," Colbert continued. "He's off message. He says Catholics need to stop obsessing about homosexuality, contraception and abortion. For Pete's sake, we need something to obsess about now that `Breaking Bad' is over."

MORE ON CNN: Pope Francis: Church can't 'interfere' with gays

The Catholic comedian also teased Pope Francis for taking a softer line on Catholic doctrine than previous pontiffs.

"Being Catholic is like being in the Admiral's Club of Christianity: Membership has its privileges," Colbert joked. "But if even atheists can be redeemed, what's next, Lutherans? It's madness."

MORE ON CNN: Heaven for atheists? Pope sparks debate

Colbert was speaking at the Al Smith Dinner, an annual event for Catholic charities in New York named after the the late New York governor who in 1928 was the first Catholic politician to be nominated for president by a major party.

The charity dinner is usually attended by New York's cultural elite. Last year, President Barack Obama and GOP presidential nominee Mitt Romney attended and traded zingers less than a month before the election.

This year's event was less politically charged, but Colbert quipped about another kind of election.

Dolan, the portly archbishop of New York and a rising star in the Catholic Church, came thisclose to being elected pope himself earlier this year, Colbert said.

"But he blew it in the swimsuit competition. I would have gone with the one-piece."

On Thursday night, Dolan, who was dressed in a cardinal's traditional small black cape with red trimming, looked like "a matador who's really let himself go," Colbert joked.

"Did you not see the invite? It said white-tie, not 'flamboyant Zorro."

Dolan, who is close friends with Colbert, took the ribbing lightly, saying that his father always told him that the mark of a good night is any one that ends in laughter.

Dolan also quoted Colbert from an event on Catholics and comedy led by both men at Fordham University in the Bronx last year.

"A sense of humor comes from faith, a faith that everything is in God's providential hands, a faith that frees us up to laugh."

Rocco Palmo at Whispers in the Loggia has an audio recording of Colbert's keynote address.