For the second day in a row, Capitol Hill took FBI Director James Comey to task on for the bureau’s decision to grant immunity to five of Hillary Clinton’s aides who were involved in her use of a private email server while serving as secretary of state.

Comey testified before the House Judiciary Committee on Wednesday, the day after he appeared before the Senate Homeland and Government Affairs Committee. On both days, Republicans peppered him with questions about the immunity agreements with Cheryl Mills, a State Department lawyer, and four other aides. Unlike Tuesday, however, the FBI director was the sole witness at the House hearing, making him the lone defender of the bureau against GOP criticism.

"Who authorized granting Cheryl Mills immunity?" Representative John Sensenbrenner (R-Wisconsin) asked.

"It's a decision made by the Department of Justice, I don't know at what level inside," Comey replied. "In our investigations, any kind of immunity comes from the prosecutors, not the investigators."

Mills’ "act of production" immunity was limited to the information contained on her laptop, as was the immunity granted to senior adviser Heather Samuelson. The deals were designed to protect the two women against any related “classification” disputes, Beth Wilkinson, an attorney representing both Mills and Samuelson, told the Wall Street Journal.

John Bentel, another State Department staffer; Bryan Pagliano, Clinton's former IT aide; and Paul Combetta, an employee at the company hired to manager her server, were also granted immunity deals. Pagliano refused to testify in front of the House Oversight and Government Reform Committee in mid-September, pleading Fifth Amendment protections against self-incrimination.

Today I filed a privileged report to hold Bryan Pagliano in contempt of Congress #subpoenasarenotoptionalpic.twitter.com/WkJZWu8oOq — Jason Chaffetz (@jasoninthehouse) September 27, 2016

#BREAKING House to punt contempt of Congress citation for fmr State Dept IT specialist Bryan Pagliano until Nov. Was slated for Thurs — Chad Pergram (@ChadPergram) September 28, 2016

"The FBI doesn't grant immunity to anybody, the Department of Justice is able to grant very different kinds of immunity," Comey said. "If new and substantial evidence develops a witness lied [under immunity], of course the Department of Justice can pursue it. Nobody gets lifetime immunity."

Rep. Louie Gohmert (R-Texas), a former judge, asked the FBI director why Mills and Samuelson received immunity when investigators suspected they too had mishandled classified information via email.

READ MORE: 5 times when the Clintons escaped federal charges

"You cleaned the slate before you even knew,” Gohmert said. “You gave immunity to people that you were going to need to make a case if a case was to be made.”

Comey explained that part of the process of giving the women deals included “queen for a day” procedures, where investigators spoke with Mills and Samuelson to hear what they had to say before deciding whether to grant them immunity.

Although House Oversight Committee Chair Jason Chaffetz (R-Utah) last week accused the FBI of handing out immunity“like candy” in the case, Comey said Wednesday that the immunity granted in the case was “fairly routine" and part of an "ordinary investigative process."

Comey expressed frustration that FBI agents were being criticized as biased or as having succumbed to political pressure, saying he has "no patience" for such accusations.

"You can call us wrong, but don’t call us weasels. We are not weasels," he told lawmakers. "We are honest people and… whether or not you agree with the result, this was done the way you want it to be done."

#NotWeasels may be the best hashtag ever used in an FBI story... https://t.co/jPDTcK6fNv — Jesse Kirsch (@JesseRKirsch) September 28, 2016

He was passionate in his defense of how the FBI handled the case, and hit back at its critics.

"I knew there were going to be all kinds of rocks thrown, but this organization and the people who did this are honest, independent people,” Comey said. “We do not carry water for one side or the other ‒ that’s hard for people to see because so much of our country, we see things through sides.”

“We are not on anybody’s side," he added.

Wednesday was the third time ‒ and the second day in a row ‒ that Comey has been grilled over the investigation since he announced that the FBI would not recommend that the DOJ prosecute Clinton at the beginning of July. Justice declined to bring charges the next day.