As Republicans continue reeling in the wake of a 2005 tape that surfaced of their presidential nominee crassly describing his interactions with women, House Republicans planned to huddle on a conference call Monday morning about what to do next.

Over the weekend, the list of Republican defections from Trump expanded, and there was growing concern that Donald Trump wouldn’t just sink himself in the election, but potentially drag the entire party down him.

Already, House Speaker Paul Ryan (R-WI) had very publicly uninvited Trump to attend a rally with him in Wisconsin Saturday, which was supposed to be their first campaign appearance together. While it appeared Trump’s running mate Mike Pence – a former congressman respected by party conservatives – was sticking with Trump for now, it was unclear whether House members were interested in taking that risk with so many of their elections on the line.

Beginning Friday, House conservatives began cutting their losses with Trump. Rep. Jason Chaffetz (R-UT), the GOP’s conservative oversight committee chairman, said he was taking back his endorsement. Rep. Martha Roby (R-AL), who hails from a very conservative district, said she wanted Pence to replace Trump as nominee because she just could not bring herself to vote for Trump.

According to a report in Politico, the speaker himself was weighing pulling his endorsement from Trump as well.