In the wake of a victory conference in the nation’s capital with white supremacists doing Nazi salutes and saying “Heil Victory” in honor of Donald Trump’s election, Ben Carson—the former presidential candidate turned prominent supporter—said the president-elect didn’t need to outright condemn this racist faction.

“I don’t think the president-elect can be held responsible for who supports him,” Carson said in a phone interview with The Daily Beast. “I didn’t see anybody blaming President Obama when the Black Panthers were doing things. I just don’t understand why we have to take anybody who supports someone and make them try to defend that person.”

He made it clear that he does not believe that Trump, or anyone else in the incoming administration, endorses the actions of these supporters.

But the confab of white nationalists, hosted by racist posterboy Richard Spencer, in Washington D.C.’s Ronald Reagan Building was not an anomaly. It was the culmination of a year-long celebration that ended in the election of Donald Trump. Steve Bannon, who masterminded the fourth quarter of Trump’s campaign, came from serving as CEO of Breitbart, which helped popularize the alt-right movement—a new branch of white nationalists.

Bannon has since been appointed as White House strategist, further emboldening the white-nationalist agenda.

Trump has faced a flurry of controversy for appointing Bannon , due to the Breitbart executive’s ties and encouragement of white-nationalist factions.

But from Carson’s particular lens, this is all a misunderstanding of the man.

“I’ve known him. I’ve never seen any of the negative things that people talk about. I’ve been to his home,” Carson said. “He’s a very gracious host. I have found nothing hateful or white supremacist in anything that he’s ever done.”

The conference of white supremacists elicited a half-hearted response from the Trump campaign just days after the president-elect launched into pearl-clutching hysterics over a smattering of boos when Mike Pence attended the musical Hamilton.

“President-elect Trump has continued to denounce racism of any kind and he was elected because he will be a leader for every American,” the statement from the transition team read. “To think otherwise is a complete misrepresentation of the movement that united Americans from all backgrounds.”

The U.S. Holocaust Memorial Museum on the other hand, was justifiably mortified by the weekend’s events.

“The Holocaust did not begin with killing; it began with words,” the museum said in a statement, noting that the conference took place just a few blocks from the museum itself. “The museum calls on all American citizens, our religious and civic leaders, and the leadership of all branches of government to confront racist thinking and divisive, hateful speech.”

Carson had seemingly distanced himself from these sorts of controversies—and from any potential role in President-elect Trump’s administration—just last week.

“Dr. Carson feels he has no government experience, he’s never run a federal agency,” his business manager and adviser Armstrong Williams told The Hill. “The last thing he would want to do was take a position that could cripple the presidency.” (A lack of government experience did not preclude Carson from seeking a job running the federal government.)

But on Monday, Carson took a slightly different tack.

“Well basically what I’ve said is I prefered to work as a non-governmental employee,” Carson explained. “If he feels that he truly needs me to be part of the Cabinet, I would give that serious consideration. My preference would be to stay on the outside.” When asked specifically about potential roles involving the Departments of Education and Health and Human Services, Carson said “those would be the natural areas.”

His role in the revolving door of conversations and considerations that are going into Trump’s selections for his burgeoning administration is unclear at this stage. Carson alluded to providing guidance in the process but, according to pool reports and daily conference calls, the renowned surgeon has not appeared at any meetings with Trump in recent days.

However on Tuesday, Trump tweeted that he was considering Carson for the head of Housing and Urban Development, after this story was published.

Throughout the presidential campaign, after Carson went from being a serious early contender for the nomination to a full-throated Trump backer, he would often—perhaps inadvertently—use the press as a conduit to try and temper some of Trump’s worst instincts. In previous conversations with The Daily Beast, he said that Trump should not call Hillary Clinton a “bigot,” and that her faith was not a matter for campaign mudslinging.

Carson did not make any similar semi-veiled condemnations in Monday’s conversation—instead he provided a defense for Trump’s selected Cabinet appointments, his plan for a tour of states he won in the election, and his appointment of Bannon.

Trump has gone silent in the two weeks since his shocking win (outside of criticizing Broadway musicals and television shows), but emphasized in his first remarks as president-elect that night that he wanted to unify the fractured nation. Asked whether it is incumbent upon the president-elect to speak to the people who are marching up to his namesake tower on a near-nightly basis, Carson suggested that it is not Trump’s responsibility; at least for now.

“I would also ask them the question ‘To what end?’ Carson said, referring to protesters who have mobilized in numerous cities across the country. “What do they expect to accomplish?”

He said that some of the angst stems from the fact that “we have a whole generation now that has grown up in a culture of no real punishment for doing bad things,” only “a little time out” and “everyone gets a trophy.” And Carson was quick to say that Trump, the subject of the protests, is not the person currently occupying the Oval Office.

“I think that is really the job of the president” not the president-elect, Carson said when asked about speaking to the protesters and their requests. “When he becomes president, I’m sure he will have plenty to say about that. We do still have an elected leader of this country. You [Obama] should step up and lead.”

During a joint press conference with German Chancellor Angela Merkel in Germany last week, Obama said he “wouldn’t advise them to be silent.”

As the final pieces get put into place in the Trump administration, Carson may find himself, willfully or otherwise, on the outside looking in. But he’s happy to step up to the bat and defend it either way.

UPDATE, 12:34 p.m. ET: This story has been updated to include President-elect Trump’s Tuesday afternoon tweet about Dr. Carson.