 -- The chairman of the House Intelligence Committee, Devin Nunes, R-Calif., will not step aside from his committee's Russia investigation, according to a spokesman, despite a growing chorus of lawmakers calling on him to do so.

The calls for recusal come after it surfaced that Nunes, who served on the Trump transition team executive committee, visited the White House grounds last week to meet a source the day before publicly sharing details about surveillance that "inadvertently collected" information on associates of President Donald Trump.

Leading the calls for Nunes to step aside in the probe was his Democratic counterpart on the Intelligence Committee, Rep. Adam Schiff, D-Calif., House Minority Leader Nancy Pelosi and Sen. Chuck Schumer, D-N.Y.

"I believe that the chairman should recuse himself from any further involvement in the Russia investigation, as well as any involvement in oversight of matters pertaining to any incidental collection of the Trump transition, as he was also a key member of the transition team," said Schiff in a statement Monday.

"This is not a recommendation I make lightly, as the chairman and I have worked together well for several years; and I take this step with the knowledge of the solemn responsibility we have on the Intelligence Committee to provide oversight on all intelligence matters, not just to conduct the investigation."

Pelosi said Nunes "has a serious responsibility to the Congress and to the Country. Chairman Nunes’ discredited behavior has tarnished that office." Last week, she called Nunes a "stooge of the president" and said he was "deeply compromised and he cannot possibly lead an honest investigation."

And earlier in the day, Schumer became the highest-profile Democrat to call for Nunes' resignation while speaking on the Senate floor.

"If Speaker [Paul] Ryan wants the House to have a credible investigation, he needs to replace Chairman Nunes," said Schumer.

Nunes raised eyebrows when he held a press conference and then briefed the president, without making members of his committee aware, that members of the Trump transition team, and possibly the president himself, had been "incidentally" after the election. Though he maintained that the intelligence gathering was conducted legally, it prompted Trump, who claimed President Barack Obama '"had my 'wires tapped'" Trump Tower, to say he felt "somewhat" vindicated.

In an interview on CNN's "The Situation Room," Nunes argued that the visit to the White House grounds was commonplace and part of an investigation into the unmasking of Americans in intelligence reports that began before Trump's wiretapping claims.

"I had been working this for a long time with many different sources and needed a place that I could actually finally go because I knew what I was looking for and I could actually get access to what I needed to see," said Nunes, adding later, "It wasn't at night ... nobody was sneaking around, all it was was just a place where I had to go to be able to review this information."

Asked whether he could've used a Sensitive Compartmented Information Facility (SCIF) at a different location to view the materials and avoid the appearance of impropriety that accompanied his White House trip, Nunes said he could not because Congress had "not been given this information." SCIFs are specifically used to view or discuss classified information in surroundings that are otherwise unsecured.

On the Republican side, Sens. Lindsey Graham, R-S.C., and John McCain, R-Ariz., had questions of their own.

"I just think he needs to explain what he did, who he talked to," said Graham. "Schiff to me is talking more like a prosecutor and Nunes has been acting like a defense attorney. The bottom line is, we’re hoping they can put it together in the House. We hope they can get back on track."

McCain said he "honestly [doesn't] know what to make of" the situation with Nunes, but said he wants to know, "What brought him to the White House? Who did he see? What was the information? Just a few complicated questions like that."

After first noting the incidental collection of information about Trump associates last week to the press, and then briefing the president on the matter, Nunes apologized for not consulting the Intelligence Committee first. Nunes said he made the decision to go public first in the midst of a busy day and that it was a "judgment call."

On Friday, a spokesman for Nunes walked back some of his comments, telling ABC News that the congressman doesn't know "for sure" if the Trump associates cited were directly surveilled.

ABC News' Ali Rogin and Benjamin Siegel contributed to this report.