Breaking his silence, Speaker Paul Ryan (R-Wis.) on Friday called President Trump's "shithole countries" remarks "very unfortunate" and "unhelpful," noting that he himself was a descendant of Irish immigrants who had faced prejudice and hostilities when they first moved to America.

The Speaker said he had read press accounts Thursday night about Trump's immigration meeting with lawmakers in which he reportedly complained about restoring protected status for immigrants from "shithole countries" like Haiti, El Salvador and some African nations.

"The first thing that came to my mind was very unfortunate, unhelpful," Ryan said at a WisPolitics event in Milwaukee. "But you know what I thought of right away? I thought of my own family.

"My family, like a whole lot of people, came from Ireland on what they called coffin ships and came here and worked the railroads. The Irish were really looked down upon in those days," Ryan said, detailing how his ancestors emigrated from Ireland and ended up in Janesville, Wisconsin.

"I hear all these stories from my relatives about 'Irish need not apply.' [The Irish] could basically get constructions jobs, cops or firefighter jobs. And James and Catherine Ryan came over and literally worked the railroad until they had enough money to buy a farm, which happened to be outside of Janesville, Wisconsin.

"Then their son, my great-grandfather, started a railroad business with horse plows and it's an earth-moving business which to this day is run by my cousins. It is a beautiful story of America, and that is a great story."

"That is a story we have today. That is a story we had yesterday. And that is what makes this country so exceptional and unique in the first place," Ryan went on. "So I see this as something to celebrate and I think it's a big part of our strength, whether you are coming from Haiti - we've got great friends from Africa in Janesville who are doctors, who are just incredible citizens."

So far, Ryan is the only member of GOP leadership to criticize Trump's remarks, roughly 20 hours after The Washington Post reported on what was said in the Oval Office meeting. But Ryan's reaction is much softer than other leading figures in the party.

Rep. Mia Love (R-Utah), who is Haitian American, called on Trump to apologize for remarks that were "unkind, divisive, elitist, and fly in the face of our nation's values." Michael Steele, who was the first African-American chairman of the Republican National Committee, said it's "incontrovertible" evidence Trump is a racist.

Trump has denied he made the vulgar comment, saying he simply used tough language to reflect his position that the United States should change its immigration policies, and said Haiti is a poor country.

But Sen. Dick Durbin (D-Ill.), who attended the meeting, said Trump did make the "hateful" remarks. Two Republicans in attendance, Sens. Tom Cotton (Ark.) and David Perdue (Ga.), said they did not "recall" Trump making the remark.

Sen. Tim Scott (R-S.C.) said the senior senator from his state, Republican Lindsey Graham, who also participated in the Trump meeting, told him the "shithole countries" comment was "basically accurate."

Ryan's top deputy, Majority Leader Kevin McCarthy (R-Calif.), also attended the Oval Office meeting, but has had no comment so far.

The firestorm over Trump's remarks complicate already delicate immigration negotiations between Democrats and Republicans. The two sides are struggling to reach a deal on how to shield hundreds of thousands of young "Dreamers" from deportation as conservatives clamor for tougher border-security measures.

Friday was far from the first time Ryan, the highest-ranking Republican on Capitol Hill, has publicly distanced himself from Trump's controversial words or actions.

During the 2016 campaign, Trump said a federal judge could not fairly rule in a case against Trump University because of his Mexican heritage.

At an anti-poverty event outside of Washington, Ryan slammed Trump's remarks about Judge Gonzalo Curiel, calling them the "textbook definition of racist comments."

"I regret these comments that he made," Ryan said. "I think that should be absolutely disavowed."