New York (CNN) Donald Trump has never been one to shy away from speaking -- or more accurately, tweeting -- his mind.

But critics say it took him too long to publicly disavow a shockingly racist speech Saturday by a white nationalist leader whose rallying cry mirrored Adolf Hitler's.

"Hail Trump, hail our people, hail victory," Richard Spencer shouted from the podium of the annual convention for his think-tank called The National Policy Institute. Spencer calls himself the founder of the "alt-right" movement, a label that's been applied to far-right extremists advocating for white nationalism.

The scene, taking place less than a mile from the White House, was reminiscent of Nazi-era Germany, with several members of the audience cheering with the straight-arm Hitler salute.

At times speaking in German, Spencer's 30-minute speech included the unmistakable marriage of Neo-Nazi hate and Trump's campaign slogan.

"It is only normal again when we are great again," Spencer said.

A Trump transition spokesman released a short media statement Monday evening, but it took Trump until Tuesday to publicly disavow the group in his own words. And it came only when pressed in a meeting with New York Times reporters, editors and executives,

But Oren Segal, director of the Anti-Defamation League, says Trump needed to do it sooner.

"There seems to be a pattern in the Trump administration of waiting until the last moment. And we just don't have the luxury for that. When there are Nazi salutes in D.C., it's important to condemn it at the moment," Segal said.

Segal referred to Trump's weekend tweets criticizing the cast of the Broadway show "Hamilton" when Vice President-elect Mike Pence attended.

"If you have the time to tweet about the theater, you should have the time to tweet about a spate of hate crimes and Neo-Nazis in Washington D.C.," he said.

Similar criticism was lobbed at Trump during the primaries when he hesitated before disavowing David Duke and the KKK on CNN's The Lead with Jake Tapper.

"I know nothing about David Duke, I know nothing about white supremacy," Trump told Tapper.

Republican National Committee RNC communications director Sean Spicer came to Trump's defense in an interview with Wolf Blitzer on CNN's "The Situation Room."

"When is it going to be enough?" Spicer said. He added, "He has said that he refutes that, that's not the kind of movement he wants the support or see energized by him. That's not the kind of people he wants to be associated with. I don't know how many more times Donald Trump can make his position clear, but he's been crystal-clear for a long time and over and over and over again with every tweet, utterance, you name it, Facebook post, he has to somehow respond to it. He has made his position clear. It's time that we accept that position and move on. That's not his focus."

Criticism of Bannon

The hiring of Steve Bannon as chief strategist in the Trump White House has only intensified the criticism of Trump on this issue.

While Bannon once reportedly bragged to Mother Jones that his website, Breitbart.com, was the "platform for the alt-right," he later told the Wall Street Journal that he has "zero-tolerance" for those anti-Semitic tones.

Trump said Tuesday that "Breitbart is just a publication. They're certainly much more conservative, but Breitbart is really a news organization that's become successful."

Regarding Bannon, Trump said, "I've known Steve Bannon a long time. If I thought he was a racist, or alt-right ... I wouldn't even think about hiring him." He later added: "I think it's very hard on him. I think he's having a hard time with it. Because it's not him."

Former Breitbart spokesman Kurt Bardella told CNN the issue will continue to cause problems for Trump.

"(Bannon) has said, Breitbart is the alt-right platform, these are the people that they have played to, that they have tried to motivate to be the base of Donald Trump's election. And it's just going to be more of this."

Spencer threatens to spread his message

Spencer's think tank used to be a small, obscure extremist group operating mostly on the Internet, but its new confidence is apparent. Spencer even told The Washington Post that he has plans to give speeches at college campuses in hopes of gaining more supporters.

But Spencer's message is white supremacism, anti-Semitic, anti-immigration and neo-Nazi. He's been banned from parts of Europe for trying to organize there.

"America was until this past generation a white country, designed for ourselves and our posterity," Spencer told the crowd. "It is our creation, it is our inheritance and it belongs to us."

Video of the event is so alarming, the national Holocaust museum in Washington wrote a letter in response, warning that "The Holocaust did not begin with killing, it began with words," and comparing Spencer's words to Hitler's.

At the Times meeting, Trump said he wanted to look into the reason why Spencer's group feels so energized by his campaign.

"It's not a group I want to energize, and if they are energized, I want to look into it and find out why," he said.

During the primary season, a self-described "white advocate" explained it to CNN, saying Trump's anti-immigrant campaign promises were appealing to their message.

"He is appealing to people who grew up in the United States and don't want to end their days in what turns out to be an outpost of Mexico or Vietnam or Guatemala or Haiti," Jared Taylor told CNN's Drew Griffin.

The Southern Poverty Law Center put out a statement Tuesday saying it should be no surprise that Trump's campaign rhetoric has given people permission to say these things out loud.

"The reason he's energized the alt-right is simple: He's been playing its tune from the day he announced his campaign and called Mexican immigrants rapists. His proposal to ban Muslim immigration, his talk of mass deportations, his attacks on 'political correctness,' it's all been music to the ears of a movement that envisions a white America -- and that's exactly the America that the 'alt-right' wants to see."