The House Democratic sit-in calling for a vote on gun legislation ended on Thursday afternoon, with lawmakers vowing to take up the cause once more after the July 4 recess.

But if the Democrats choose to continue pursuing what has been their chief goal so far – a bill that would bar those on the federal government’s terrorist watch list from purchasing firearms — they will face opposition from some of their traditional allies.

House Speaker Paul Ryan cited the ACLU’s stand against the bill at a press conference Thursday. “In this country, we do not take away people’s constitutional rights without due process. This is not just Republicans saying this. It’s groups like the ACLU who are saying this,” he said.

The ACLU opposes the Democratic proposal on the grounds that the government’s watchlisting system “is error-prone and unreliable because it uses vague and overbroad criteria and secret evidence to place individuals on blacklists without a meaningful process to correct government error and clear their names.”

The Council on American Islamic Relations (CAIR), the nation’s leading Muslim American civil rights group, also came out against the proposal on Wednesday.

“It would seem the Senate is willing to only apply constitutional limitations on the American Muslim community, which is disproportionately impacted by federal watch lists,” CAIR wrote in a statement, adding that it does, by contrast, support gun-control measures like expanded background checks and lifting the ban on federal research on gun violence.

Although some Republicans are citing due process concerns to oppose the no-fly-no-buy proposal, few are actually supporting legislation to reform the watchlist. Rep. Justin Amash, R-Mich., introduced a bill that would prohibit the Department of Homeland Security from barring anyone not convicted of a crime from flying on an airplane or boarding a cruise ship. It has no cosponsors. A more modest bill introduced in 2013 to speed up the redress process for being placed on the list had only two sponsors, and no Republicans. Not one Republican joined a 2014 letter to DHS protesting flaws in the listing process — although the Democratic signatories to that letter now evidently think it’s good enough to use for barring gun sales.