(CNN) Republicans in Congress are sharpening their criticism of Robert Mueller and his team of prosecutors, a sign that the special counsel investigating the President Donald Trump's campaign is increasingly losing GOP support on Capitol Hill.

In the House and the Senate, GOP lawmakers have been irked about a series of revelations -- most notably the removal of an FBI agent from Mueller's team for sending anti-Trump texts -- saying that the special counsel needs to get rid of prosecutors with Democratic leanings in order to carry through an impartial investigation.

Their comments could give the White House new fodder to fight an investigation the President has deemed a "witch hunt," and renew a push to name a second special counsel to investigate potential wrongdoing by Hillary Clinton, and other Justice Department concerns connected to the Mueller investigation. And it could undermine a new effort by a bipartisan group of senators to draft a compromise bill aimed at protecting the special counsel.

Some Republicans who have supported Mueller to this point say their patience is wearing thin.

"The last couple of weeks have not been good," said Rep. Trey Gowdy, the House Oversight Chairman and member of the House Intelligence Committee, who earlier this year urged Republicans to give Mueller "a chance to do his job."

"That group I'm in -- that small group (of Republicans) that has supported Mueller and resisted calls for a special counsel -- is getting smaller," Gowdy told CNN.

"The fact that you can't find prosecutors that don't have an 'I'm with Her' T-shirt on to staff your special counsel office -- it's just tone deaf," Gowdy added.

Reports this past summer found that several of the lawyers on Mueller's team donated to Democrats.

Texas Sen. Ted Cruz, a conservative who serves on the Senate Judiciary Committee, said he has serious questions about whether Mueller's probe can be impartial.

"I think that some composition of the lawyers working on the special counsel investigation raises serious questions about the investigation's ability to be impartial," Cruz told CNN. "Far too many of the special counsel prosecutors have significant records contributing to Democratic politicians -- and representing partisan Democratic clients. That is highly troubling."

Senate Majority Whip John Cornyn, a member of the judiciary and intelligence panels, called on Mueller to clean house and rid his team of any prosecutor who has backed Democrats.

"I would eliminate people with those sorts of backgrounds from the team because I think they raise suspicions that this is not a fair and independent and objective investigation and prosecution," Cornyn said.

The criticism is bound to come sharply Wednesday when Rod Rosenstein, the deputy attorney general, testifies before the House Judiciary Committee, given that he oversees the Mueller probe.

A small group of Republicans have been going after Mueller's probe for months, but the Republican criticism toward Mueller ratcheted up after reports emerged that FBI agent Peter Strzok was removed from Mueller's team over sending anti-Trump text messages during the presidential campaign. Details of the actual texts, which included referring of then-presidential candidate Trump as an "idiot" and "d*uche," were released to lawmakers Tuesday night.

A growing number of conservatives are questioning the handling of the Clinton email investigation, with some saying that the Justice Department should name another special counsel to look into the inquiry that cleared Clinton of any criminal wrongdoing.

The calls for a second special counsel amplified this week after Fox News reported that Bruce Ohr was demoted over 2016 meetings with Fusion GPS founder Glenn Simpson and Christopher Steele, the ex-British intelligence agent who assembled the dossier on Trump and Russia. Fox also reported that Ohr's wife, Nellie Ohr, worked for Fusion GPS during the 2016 election.

Jay Sekulow, a member of Trump's legal team, said Tuesday that another special counsel should be appointed to investigate Ohr and Justice Department ties to Fusion GPS.

Moreover, GOP lawmakers have bashed the Justice Department for not providing Congress with enough information about the ties between the FBI and Fusion GPS, an opposition research firm that commissioned the dossier last year containing allegations of connections between Trump and Russia.

Marc Elias, the Democratic lawyer who was retained by the Clinton campaign and Democratic National Committee and whose firm paid for the research, is scheduled to meet with the House Intelligence Committee on Wednesday, according to a source familiar with the matter.

On Tuesday evening, a half-dozen conservative Republicans went to the House floor to slam Mueller's probe.

"Right now, the investigations into Donald Trump and the prior investigation into Hillary Clinton have been infected with the virus of severe bias," said Rep. Matt Gaetz, a Florida Republican. "Hillary Clinton went under investigation for the mishandling of classified information, and also her dealings through the Clinton Foundation, was essentially investigated by her own fan club. Meanwhile, Robert Mueller obtained his team by fishing in the never-Trump aquarium."

Some Republicans have also called for another special counsel to investigate a major uranium deal with a Russian state-owned energy company that was approved by the Obama administration and whether Clinton Foundation donors had any influence in its approval.

Sen. Chuck Grassley, the chairman of the Senate Judiciary Committee, tweeted in October that the Justice Department should name a special counsel to investigate the uranium deal.

Asked if there should be another special counsel named, Grassley told CNN Tuesday: "I'll have to refer you to my tweet."

Sen. Lindsey Graham, a member of the Senate Judiciary Committee tweeted that "it's long past time" for the special counsel to investigate "the Clinton email scandal" and other matters.

Sen. Thom Tillis, a North Carolina Republican who also serves on Senate Judiciary Committee, said the jury is still out on Mueller's probe, but added: "I haven't drawn a firm conclusion but I do think that there are facts that have come up over the past couple of weeks that need to be tracked down that the FBI investigators seem to be biased."

Tillis is one of four senators who have been working to craft a bipartisan bill that would attempt to protect the special counsel.

There are two bipartisan bills in the Senate aimed at protecting the special counsel, including Tillis' legislation, and the senators are working on combining the measures to move one bill forward in the Senate.

Some Republican lawmakers say they are still behind Mueller's investigation, such as Sen. Susan Collins, a Maine Republican on the Senate Intelligence Committee.

"My advice to the President would be to tell his staff and associates to fully cooperate with Robert Mueller's investigation and to not have any comment on it beyond that," Collins said.

Collins also downplayed the need for legislation to protect Mueller.

"I haven't seen the need for that because I really don't think the President is going to fire Mr. Mueller," she said. "But certainly the introduction of those bills sends a strong signal."