The cascade of controversies buffeting the White House is starting to rankle high-profile Republicans, who are voicing concerns about how the daily drama is impacting the GOP agenda on Capitol Hill.

Even as House Speaker Paul Ryan tried Wednesday to tamp down the furor over the latest accusation -- that Trump asked James Comey to shutter part of the FBI's Russia probe -- other Republicans were showing their exasperation.

"We cannot have this constant chaos and serious questions being raised virtually every single day," Sen. Susan Collins, R-Maine, said. "It is distracting from the ability to work on certain issues like health care reform, and the administration needs to get its house in order."

Most Republicans on the Hill still are not going along with Democratic demands for a special prosecutor, but a handful are. Even more are opening the door to further inquiries into the latest Comey accusation.

Part of this reflects a desire by Republicans not to rush to judgment over the claims in a memo none of them have seen. According to The New York Times, Comey said in a February memo that Trump asked him to end the investigation into former national security adviser Michael Flynn.

"We need the facts," Ryan said, echoing comments from Rep. Trey Gowdy, R-S.C., and others.

The White House maintains that Trump did not seek to end any investigations.

But Republicans' fresh inquiries suggest the controversy will continue to play out, as they ask to view the memo and hear from Comey. House Oversight Committee Chair Rep. Jason Chaffetz, R-Utah, asked overnight for documents including the reported February memo. Chaffetz tweeted he's ready to subpoena the information if necessary.

Colorado Republican Rep. Mike Coffman tweeted: “We need the memos, Comey should testify and I still believe that a special prosecutor should be named.”

Rep. Mike Gallagher, a Wisconsin Republican in his first term, tweeted: “For the purpose of transparency, the White House should share a transcript of the meeting with the House and Senate intelligence committees.”

And regarding a separate report -- also disputed by the White House -- that Trump shared classified intel with the Russians, Republican Virginia Rep. Barbara Comstock called for "immediate classified briefings."

But as Republicans exercise their oversight responsibilities, they're openly voicing concern about what it means for the GOP agenda.

One senior House GOP source told Fox News that the mood at the House Republican Conference meeting Wednesday was "melancholy."

Chairman of the Senate Foreign Relations Committee Bob Corker, R-Tenn., said the White House is "in a downward spiral right now” and the Trump administration has “got to figure out a way to come to grips with all that’s happening.”

Sen. John McCain, R-Ariz., also got in on the Trump bashing and invoked “Watergate.”

Senate Majority Leader Mitch McConnell, R-Ky., told Bloomberg Live that "we could do with a little less drama from the White House" so lawmakers can focus on tax reform, rolling back red tape and repealing ObamaCare.

The Comey controversy followed a high point for Hill Republicans, after the House revived the struggling ObamaCare repeal bill and passed it, sending it to the Senate. The biggest congressional win before that was Neil Gorsuch's confirmation to the Supreme Court.

Ryan and his deputies tried Wednesday to reset focus on tax reform and other priorities.

"If we're serious about letting Americans keep more of their paychecks, if we're serious about bringing jobs back home, bringing manufacturing back home, then we have got to reform this tax system and seize this moment," Ryan said Wednesday.

Whether Republicans can dust themselves off may depend on what the Comey memos and other documents show.

Lara Brown, director of George Washington University’s political management program, told Fox News she expects more fallout as the midterm elections approach. She predicted more Republicans will pull away from the Trump White House around October, giving health care legislation low odds and saying tax reform is "dead in the water."

“In the fall, each individual politician will make their strategic calculations if they can attach themselves to the president or distance themselves,” she said. “We’re going to see increasing numbers of Republican partisans moving away.”

Fox News' Chad Pergram contributed to this report.