A weekend-long backlash over a decision to invite alt-right provocateur Milo Yiannopoulos to the Conservative Political Action Conference ended Monday with an abrupt reversal.

Yiannopoulos, an editor at Breitbart News and a frequent supplier of racist and sexist rhetoric, will no longer speak at the annual confab, a decision that appeared to catch him off guard.

"I haven't heard any indication that they are reconsidering," Yiannopoulos told CNNMoney early Monday afternoon.

That changed less than an hour later, with the sponsor of the conference, the American Conservative Union, announcing that it had rescinded Yiannopoulos' invitation.

The reason for the turnabout: A pair of video clips that surfaced Sunday in which Yiannopoulos appears to be speaking sympathetically about sex with young boys and cracking a joke about his own sexual encounter with a Catholic priest as a child.

"We continue to believe that CPAC is a constructive forum for controversies and disagreements among conservatives, however there is no disagreement among our attendees on the evils of sexual abuse of children," ACU president Matt Schlapp said in a statement.

That statement was apparently released to the public before it was seen by Yiannopoulos. When asked to confirm that his invitation to CPAC had been rescinded, Yiannopoulos told CNNMoney it was news to him.

"If so," he said in an email, "I haven't heard that."

Later Monday, Simon & Schuster announced that it was canceling publication of Yianopoulos' forthcoming book "Dangerous."

CPAC's decision marked the culmination of a firestorm that erupted Saturday, when the ACU announced Yiannopoulos' appearance at the conference. Many prominent conservative pundits provided the loudest voices of opposition to the decision. By Sunday, following the release of the two incendiary videos, the chorus of dissent had grown nearly deafening.

Bill Kristol, the editor-at-large for the conservative Weekly Standard, said the invitation to Yiannopoulos was "despicable."

Related: Milo Yiannopoulos snags book deal with Simon & Schuster imprint

Ned Ryun, a board member for the ACU, likewise objected to the decision. He said on Monday morning that members of the board were not consulted on the decision.

"While I'm all for free speech, there is such a thing as vile, hateful speech that does not deserve a platform," Ryun tweeted.

Jonah Goldberg, a senior editor at the conservative National Review, described Yiannopoulos as "a promoter and apologist for the 'Alt Right,' a white supremacist coalition that seeks to be the alternative to mainstream conservative movement."

"That in itself should be the only red flag CPAC needs,"

Following the decision to disinvite Yiannopoulos, Goldberg responded with something close to an eyeroll.

"Apparently the racism and anti-Semitism wasn't a deal breaker," Goldberg said.

Rich Lowry, the top editor at the National Review, said it was "a colossal misjudgment to invite him."

"He's not a conservative, and in fact wants to overthrow Reagan conservatism, besides his other obvious failings," Lowry said. "Now having disinvited him, CPAC looks like the censor--the worst of both worlds."

Related: Milo Yiannopoulos: 'I just want to burn it down'

In one of the videos, Yiannopoulos defended sexual relationships between "younger boys and older men."

"In the homosexual world, particularly, some of those relationships between younger boys and older men — the sort of 'coming of age' relationship — those relationships in which those older men help those young boys discover who they are and give them security and safety and provide them with love and a reliable, sort of rock, where they can't speak to their parents," he said.

Yiannopoulos wrote in response that the videos were "deceptively edited" and accused establishment Republicans of trying to smear him.

"I do not support pedophilia. Period," he wrote.

Related: 10 of Breitbart's most incendiary headlines

In a phone interview on Monday, Yiannopoulos reiterated that point and said he was "guilty of imprecise language."

"My kind of dry British sarcasm and penchant for provocation could have come off as flippant to other people, and that was unintended," he told CNNMoney. "To be clear, I think it's a vile and disgusting crime, and I'm horrified that people think I believe otherwise."

Yiannopoulos said he believes his teenage experience with a member of the clergy gave him a license to discuss the matter in his own way, likening it to "gallows humor."

"As a gay man who has some experience with this in his own childhood and adolescence, I thought it was OK to talk about the subject however I wanted because I felt like I owned it," he said.

CPAC, which regularly attracts political and media luminaries on the right, will kick off on Wednesday in National Harbor, Maryland.

The event has become an essential stop for Republicans seeking public office and conservative commentators vying for a larger audience. The ACU announced Monday that President Donald Trump will speak Friday at the conference.

In some ways, the objections raised by the likes of Goldberg and Kristol harkened back to last year's Republican presidential primary, when old guard conservatives opposed Trump while right-wing upstarts like Breitbart embraced his candidacy.

Yiannopoulos said Monday that the uproar on the right over his CPAC invitation was another "chapter in that story."

It wouldn't have been Breitbart's maiden voyage at CPAC. The site's founder and namesake, Andrew Breitbart, spoke there in 2012, weeks before his death.

The outlet has seen its influence swell in the years since. Its web traffic surged to record highs last year, establishing itself as perhaps the go-to source for Trump supporters. Steve Bannon, the former Breitbart chairman, now serves as one of Trump's top advisers and will speak at the conference.

Earlier this month, the University of California-Berkeley canceled a scheduled appearance by Yiannopoulos after protests over the event turned violent.

Last year, Yiannopoulos was permanently banned from Twitter for leading a harassment campaign against "Saturday Night Live" cast member Leslie Jones.

For a while, it appeared that Yiannopoulos might survive the controversy, as Schlapp defended the decision into the evening on Sunday. But by Monday, the pressure -- particularly from those on the right -- proved too much.

"We realize that Mr. Yiannopoulos has responded on Facebook, but it is insufficient," Schlapp said in a statement. "It is up to him to answer the tough questions and we urge him to immediately further address these disturbing comments."