(CNN) House Speaker Paul Ryan ripped Donald Trump's recent remarks saying a judge presiding over a lawsuit involving his business was biased because of his Mexican heritage as "the textbook definition of a racist comment."

As he stood surrounded by community activists and top House GOP members, Ryan admitted Tuesday that the firestorm over Trump's comments was undercutting his own push to roll out a positive policy agenda for his party. He said he wouldn't defend Trump but instead would concentrate on his agenda project.

"I disavow those comments," Ryan said.

Pressed on whether he regretted his own endorsement of Trump that came last week, Ryan added, "I regret those comments he made."

"It's absolutely unacceptable," Ryan said. But he stood by by his support for the controversial business mogul. "Do I think Hillary Clinton is the answer? No I do not."

Continued GOP angst about the fallout from Trump's comments cast a major shadow over Ryan's months of work crafting a detailed agenda plan for his party.

JUST WATCHED Trump: Ryan endorsement about GOP 'winning' Replay More Videos ... MUST WATCH Trump: Ryan endorsement about GOP 'winning' 01:53

Ryan said ultimately he has more in common on the policy issues of the day with Trump and better odds of getting those proposals enacted into law.

The speaker said he has discussed the tone of the campaign the presumptive GOP presidential nominee, but he deflected questions about the GOP nominee's campaign or the need for new staff or a more disciplined message.

Ryan, who was making his third visit to the City of Hope House of Help center in an impoverished Washington neighborhood, stressed the success stories of those who have overcome addiction and other problems and become productive members of the community.

"We wanted to start with poverty because we think this sums up our case," he explained.

He said his GOP plan mirrors the work done at the center, saying that the focus of the program is to "not isolate the poor -- they are elevating the poor."