Matt Schlapp, the chairman of the American Conservative Union, was in the eye of a hurricane when we arrived at his office for a chat over ice cream.

Barely an hour earlier, Schlapp had announced that he would drop Milo Yiannopoulos from the Conservative Political Action Conference, the annual gathering his organization runs that attracts everyone on the right from President Donald Trump down to grassroots organizers.

The move came after a public backlash, much of it from fellow conservatives, over the decision to include Yiannopoulos. Critics argued that Yiannopoulos was too close to the "alt right," an online movement steeped in white nationalism that is notorious for harassing targets on social media.

Chairman Matt Schlapp speaks during the Conservative Political Action Conference in National Harbor, Maryland, in February 2015. AP

Those concerns peaked on Sunday night after a conservative group, Reagan Battalion, posted video clips of media appearances in which Yiannopoulos appeared to defend sex with young teenagers in certain circumstances.

Yiannopoulos argued that he was misinterpreted, but Schlapp decided Monday that his explanation didn't cut it and dropped him from the event.

Related: CPAC Drops Milo Yiannopoulos After Backlash

We discussed how the episode went down and what it says about the broader state of conservatism. Below is our conversation, edited for length and clarity.

Q: Could you tell me a little about how you ended up inviting Milo Yiannopoulos?

SCHLAPP: He came here and very politely asked to be part of the program, which we thought was a respectful, nice way to start the conversation.

Q: Did you know much about him before then?

Of course, I had seen his videos throughout the presidential campaign, and people in my family had seen the videos. He says quotable, salacious, interesting things. He's not politically correct in any way, and the thing we thought really was worthy of highlighting is the fact there are voices on American campuses that are just shut down. We think those voices are usually voices that stem from the center-right, and we talk about that every year at CPAC, and we think that's an appropriate thing to talk about.

Q: At what point did you realize this maybe wasn't a good idea?

SCHLAPP: Over the weekend, a video surfaced that we found offensive, disgusting, which seemed to rationalize pedophilia, and that was several steps too far. I think, without dissent, everyone in our leadership believed the right thing to do was to rescind the invitation.

Q: One member of the ACU board, Ned Ryun, was tweeting that he was upset about the Milo invitation on Sunday. Did you hear from the board right after this decision?

SCHLAPP: I can't comment on what happens when the board discusses — that's not appropriate for me to disclose. I will simply say we've gotten dozens of calls and emails and texts saying we handled the situation appropriately. I think the board feels like it was handled well.

Q: Over the weekend, even before this tape dropped, you were having a conversation online with a lot of prominent conservatives who were upset about Milo's inclusion. One argument a lot of them were making was that while Milo himself says 'I'm not a white nationalist, I'm not an anti-Semite,' he's played a role in popularizing the alt right, which seems to undeniably include a lot of those. Are they correct to be concerned about the alt right?

SCHLAPP: Sure. We are concerned about the alt right. We're going to talk about the alt right at CPAC. We're going to explain that we don't believe that's a legitimate voice in conservatism. I think people who are concerned about the alt right are properly focusing on making sure it doesn't get mainstreamed, because it shouldn't be.

Q: Were you concerned at all that having Milo might mainstream it?

Let our news meet your inbox. The news and stories that matters, delivered weekday mornings. This site is protected by recaptcha

SCHLAPP: I also don't want to allow everyone in the media to say everyone is in the alt right, which is what they try to do too often. Then it's their judgment on who is a racist and who isn't. So we had conversations with Milo about what he would talk about. He was very respectful in his approach to us. He disavowed having anything to do with the alt right, and he wanted to talk about the First Amendment on the college campus.

I would say one additional thing: Last year, at this time, many of those same conservatives were attacking me and the ACU for having Donald Trump speak. I understand their objections to Milo, but I also understand that their objections go broader. This is what tends to happen at this time of year: There are conservatives who like to attack us for either who we invite or don't invite, and after a while, I suppose, you just know that that's coming.

Q: Another concern raised was online abuse. I'm sure you know Ben Shapiro, the former Breitbart writer who has been very critical of the alt right and Milo. The Anti-Defamation League had a report last year that said there were more anti-Semitic tweets leveled against him than, I think, any other reporter in the country, which they linked to the alt right.

There was a tweet Milo sent in May after Ben Shapiro's child was born. It said: "Prayers to Ben who had to see his kid come out half-black. And already taller than he is!" Then there was a picture of a black child.

Is there any responsible way to give someone a platform if they use it like this?

SCHLAPP: The only thing I will say is that it's an important platform, and we don't condone everything that gets said by every speaker. We'll have over 100 speakers. There's only so much research you can do on a speaker, and the thing that we're more focused on is what they're going to talk about at CPAC.

We are OK having people on the stage that we disagree with. This year, we'll have mostly conservatives, but they won't be exclusively conservative. We believe that it's best to put people — even people who are provocative — in front of our attendees as long as it's done with the right tone. Let them make their decision at the end. I don't want to be the editor and censor of what everyone says.

Q: Two years ago, Phil Robertson was at CPAC for a free speech award after he was suspended from his show for remarks disparaging gays and suggesting African-Americans were happier under Jim Crow. This year, there was Milo. It's been a decade now, but Ann Coulter used a slur onstage.

The argument some conservatives make is that sometimes the message that gets out is not "we celebrate free speech," it's "as long as you upset liberals, you're OK in our book."

SCHLAPP: We totally reject that. We actually spend a lot of time going through a whole process determining what topics will be discussed. Any CPAC watcher over the last three years can see there is a much more substantive discussion on stage, not just conservative stump speeches one after the other. We believe it is an important responsibility we have to make sure the activists that come from around the country get better educated on these issues.

I feel quite bad that the decision around Milo has turned into what it has turned into when the intent was simply to focus on "why does the left shut down legitimate political dialogue?" We're going to have that conversation still.

Q: That seems to get to some of the concerns they're raising. CPAC has informative speakers. Every presidential candidate has to make a stop there. The leaders of most conservative advocacy groups are there. Is there any concern it drags them down to have some of the more outrageous speakers?

SCHLAPP: I think we try very hard not to have outrageous speakers. I think we try very hard to have speakers who speak to the political moment. If the charge is you're simply trying to get attention by having dramatic speakers, my answer would be that's not really our intent at all. If you look at most of the content at CPAC, there's more people with Ph.D's then anything else and that intentional.

Bill Maher, left, talks to Milo Yiannopoulos on HBO's 'Real Time With Bill Maher' on February 17. Janet Van Ham / AP

Q: You mentioned that this fight over Milo was a proxy for a bigger fight going on with the right. I think it's interesting we're at this moment where Republicans control everything, but there's so much hand-wringing on the right about whether this new government is conservative and whether conservatism is what it used to be. What's the state of conservatism right now?

SCHLAPP: I think we've never had so much success. We've never won so much. We've never been in a position to have this much political power. Some people can't handle that. They can't handle the success of it. They can't handle the fact that there's new aspects to it, they can't handle change, they can't handle disruption, they can't handle the fact there there's been so much change in such a short period of time. Some conservatives are traditionalists, and the idea of breaking things up and changing things is concerning to them at times.

Then there are others across the country who are so distraught at the state of things in their country that they think the only way to get conservative policy achievements is to have this great disruption. I'm not going to spin you and say this conversation isn't happening among conservatives. It's real to people across the country.

I think most people across the country voted for this and want it. They don't want it every moment of every day on every thing, but they like the idea there's dramatic change and that there's disruption. And I think there are some conservative elites who populate fancy restaurants in New York and Washington who are used to educating the grassroots on the way to look at things. And I think we've seen a complete change in that order, where I think the activists are trying to explain to those elites they're not buying it anymore. And all of that goes down at CPAC. We're happy it happens there.

Q: How do you describe Trump's worldview? Is it conservative or is it something else?

SCHLAPP: I do think he has conservative instincts. I think he had anti-Obama instincts. I think he saw Obamacare, the Obama foreign policy, the Obama economic policies as harmful to the country.

I think where conservatives were wary of Trump is he had not advocated for conservative positions for decades. He had been more meandering in his politics, including being a Democrat, including writing checks to Democrats. I have said that it's impossible to say what a candidate is until they get to office, especially someone who has never been in office before.

He's clearly put conservative around him, he's clearly taking conservative policy positions, and I think for conservatives, the first, most important pick was selecting Mike Pence, then picking Neil Gorsuch to fill this tiebreaking vote on the Supreme Court. Looking at the Cabinet. Most conservatives are pretty happy.

Q: You and your wife both served in the George W. Bush administration.

SCHLAPP: We met there! We met in Richard Nixon's old office.

Q: Some of the fiercest Trump critics within the party are former Bush officials, especially on national security. You've been supportive of Trump much earlier than a lot of Republicans. Has it been tough bridging the gap between these worlds?

SCHLAPP: I was maybe the earliest person who was constructive toward Trump who served at a high level in the Bush administration. I took a lot of guff from my former Bush colleagues who didn't see what was happening in the country. I'm not smarter than they are or a better person than they are. I just traveled a lot around the country.

I'm not telling you I could have predicted Donald Trump would get the nomination, but it was clear to me the person who would win the nomination would be an outsider who would not run the same old campaign, who would not use phrases that were careful and well conceived, smooth and inoffensive.

Q: A big part of George W. Bush's appeal was that even if you didn't like his politics, he was seen as a good, God-fearing man, married to the same woman the whole time, who really believed in what he stood for. Donald Trump is poised to enact a very socially conservative agenda, but personally it's hard to associate him with those characteristics, don't you think?

SCHLAPP: I think what Republican and conservative voters realized is they kept picking the cookie-cutter candidate.

I'll use Mitt Romney as an example. Seems like a very decent man, seemed to have a beautiful marriage — picture postcard family. But that didn't translate into fighting for the issues they care about.

People in the press corps and others were shocked someone who's been divorced a couple of times, someone who has kids from other marriages, someone who has been on the Howard Stern show or whatever example you want — how could he be someone they support? The reason is they felt led on by these other candidates who personally had the characteristics they wanted to see, but they didn't have that fight. And after eight years of Obama one thing is clear: Conservatives want a fighter.

Q: It's sort of weird, at this point, the idea of a CPAC where you aren't in the opposition. How does it change the character of the event?

SCHLAPP: It's no longer about what we would do if we had political power. It's about what we have to do now that we have political power. So it's about the reality of what it takes to pass something through Congress, to get a regulation stopped or pushed through, to pick judges who are strict constitutionalists. It's a much more practical conversation.

P.J. O'Rourke says that conservatives really hate government and every couple of years we put them in charge and then we're reminded how much we really hate government. We're not always necessarily great at the task of running government. We're the anti-government party. It actually makes some sense we're not so good at that. But you got to have basic competence in how you run the government, even in how you reduce its effectiveness in people's lives.