WASHINGTON (AP) — Top Republicans remained largely silent after President Donald Trump said over the weekend that four women of color in Congress should “go back” to the countries they came from. By Monday, some in the party were speaking up.

Sen. Tim Scott, R-S.C., the only black Republican senator, said Trump made “unacceptable personal attacks” and used “racially offensive language.”

Sen. Susan Collins, the Maine Republican who faces a potentially tough reelection alongside Trump in 2020, called the president’s comment “way over the line.” While Collins said she disagrees “strongly” with many of the views of the “far-left” members of the House Democrats, she said the president’s tweets should be removed.

Time and again Republicans in Congress have been unwilling to fully confront Trump, more wary of enduring a backlash from the president.

“There is no excuse for the president’s spiteful comments — they were absolutely unacceptable and this needs to stop,” said Sen. Lisa Murkowski, a moderate Republican from Alaska.

Still, the criticisms of Trump were coming initially from rank-and-file lawmakers, not Republican leaders. House Minority Leader Kevin McCarthy and Senate Majority Leader Mitch McConnell had yet to comment.

Time and again Republicans in Congress have been unwilling to fully confront Trump, more wary of enduring a backlash from the president than standing up for what have been considered the norms of political rhetoric.

It’s similar to past instances, most starkly when Trump said there were “good people” on both sides of a white supremacist neo-Nazi rally in Charlottesville, and Republicans gave mixed comments.

“You ought to be a little more concerned about the Republican brand,” said Jeff Flake, the former Arizona senator and outspoken Trump critic, in an interview with The Associated Press.

John Kasich, the former Ohio governor and one-time Trump rival for the White House, said “We all, including Republicans, need to speak out against these kinds of comments that do nothing more than divide us and create deep animosity — maybe even hatred.”

On Sunday morning, Trump tweeted that the “‘Progressive’ Democrat Congresswomen” should “go back” and help fix the “broken and crime infested” countries they came from and then come back and “show us how it is done.”

Trump appeared almost certainly to be referring to the four new lawmakers — Rep. Alexandria Ocasio-Cortez, D-N.Y., Ilhan Omar-D-Minn., Ayanna Pressley, D-Mass., Rashida Tlaib, D-Mich. — who are among the most outspoken against Trump administration policies and have made headlines in their ongoing divisions with House Speaker Nancy Pelosi. They all support impeachment.

Three of them were born in the United States and Omar is a refugee from Somalia.

By Monday, as the White House sought to smooth Trump’s tweets, the president doubled-down and said it was up to the women to apologize for “their horrible & disgusting actions!”

One Republican ally of Trump, Sen. Lindsey Graham of South Carolina, sought to nudge Trump to focus on their policies, even as he also called the women “anti-American.” But others took a stronger line.

Rep. Will Hurd of Texas, one of a handful of African American Republicans in Congress, said Trump’s tweets against the women lawmakers are “racist and xenophobic.”

“It’s behavior that’s unbecoming of a President of the United States,” Hurd told CNN.

Pennsylvania GOP Sen. Pat Toomey said in a statement, “President Trump was wrong to suggest that four left-wing congresswomen should go back to where they came from.” Toomey said their citizenship “is as valid as mine.”

READ MORE: How Trump talks about race

The senator said while he couldn’t disagree more with the four congresswomen’s views on “virtually every policy issue… they are entitled to their opinions, however misguided they may be.”

In Michigan, two Republican congressmen who serve alongside Tlaib, criticized Trump.

Rep. Fred Upton, serving his 17th term, told Michigan-based radio host Michael Patrick Shiels that Trump’s comments were “really uncalled for” and “very, very disappointing.”

Fellow GOP Rep. Paul Mitchell, whose district heavily backed Trump in 2016, tweeted: “We must be better than comments like these. I share the political frustrations with some members of the other party, but these comments are beneath leaders.”

Associated Press writers Alan Fram, Matthew Daly and Laurie Kellman in Washington; David Eggert in Lansing, Mich., and Meg Kinnard in Columbia, S.C., contributed to this report.