Jim Bourg / Reuters Speaker Paul Ryan (R-Wis.) is trying to straddle the line between supporting Donald Trump for president while denouncing virtually everything Trump says. Good luck with that!

WASHINGTON ― The fallout over Donald Trump’s attacks on the family of a Muslim American soldier continued Monday as military vets in Congress urged Speaker Paul Ryan (R-Wis.) to drop his endorsement for his party’s presidential nominee.

“As veterans who previously served on active duty, we are horrified by Republican presidential nominee Donald Trump’s slander of parents whose son died serving our country,” reads a letter to Ryan, signed by Democratic Reps. Ted Lieu (Calif.), Ruben Gallego (Ariz.) and Seth Moulton (Mass.).

“Your continued endorsement of Mr. Trump’s hateful, bigoted and sexist vision threatens the integrity of the House of Representatives in which we serve,” they wrote. “We were heartened by your first instinct, which was not to endorse Mr. Trump. We respectfully request that you follow what we believe your heart is telling you and withdraw your endorsement of him now.”

Their letter comes after last week’s moving speech at the Democratic National Convention by Khizr Khan, who spoke of the heroism of his son, U.S. Army Capt. Humayun S.M. Khan, who was killed in Iraq while protecting his troops. Khizr Khan denounced Trump and his proposed ban on Muslims, asking if the real estate mogul has even read the Constitution. Trump responded by criticizing Khan for saying he has “sacrificed nothing,” claiming Khan “has no right” to say “inaccurate things” in front of millions of people. He also questioned why Khan’s wife Ghazala, who stood by his side during his speech, didn’t speak, suggesting she wasn’t allowed to talk because Islam oppresses women.

That hasn’t gone over well with anyone, including Republicans.

Ryan spokeswoman AshLee Strong declined to say Monday if Ryan would pull his support for Trump over the flap. Instead, she pointed to a statement Ryan issued over the weekend, in which he says Capt. Khan’s sacrifice should be honored but says nothing about Trump himself.

“Many Muslim Americans have served valiantly in our military, and made the ultimate sacrifice. Captain Khan was one such brave example,” reads Ryan’s statement. “His sacrifice ― and that of Khizr and Ghazala Khan ― should always be honored. Period.”

There are dozens of military veterans in Congress, and for those who are Republican, Trump’s latest flap makes it even more difficult for them to stand by their party’s nominee. The Huffington Post reached out to ten GOP lawmakers who served overseas, randomly selected, to see if they still endorse Trump for president ― or if they plan to formally endorse Trump at all. Only a few responded.

“The Congressman has yet to endorse,” said Maura Gillespie, a spokeswoman for Rep. Adam Kinzinger (R-Ill.).

“Rep. Coffman has not endorsed Trump,” said Cinamon Watson, a spokeswoman for Rep. Mike Coffman (R-Colo.). “In fact, Rep. Coffman has voiced grave concern about Mr. Trump’s policies and his tone.”

That’s not the full story, though. A Coffman aide said in February the congressman would “obviously” support Trump over Clinton.

“Of course Hunter is still on board with Trump,” added Joe Kasper, a spokesman for Rep. Duncan Hunter (R-Calif.). “But I think this is a real learning lesson for Trump in that he’s got to be more careful when discussing points of such sensitivity. In this case, he should have acknowledged the sacrifice and separated it from the broader policy issue, but he’s still right on the problem related to immigration and I have no doubt that he believes anyone who makes the ultimate sacrifice is the truest of heroes.”

Aides to Republican Reps. Doug Collins (Ga.), Brad Wenstrup (Ohio), Joe Heck (Nev.), Martha McSally (Ariz.), Scott Perry (Pa.) and Republican Sens. Joni Ernst (Iowa) and Tom Cotton (Ark.) did not respond to a request for comment.