Washington (CNN) It has been something of a mystery, the whereabouts of House Intelligence Committee Chairman Rep. Devin Nunes on the day before his announcement that he saw information suggesting that communications of then-President-elect Donald Trump and his advisers may have been swept up in surveillance of other foreign nationals.

The California Republican confirmed to CNN in a phone interview Monday he was on the White House grounds that day -- but he said he was not in the White House itself. (Other buildings, including the Eisenhower Executive Office Building, are on the same grounds.)

Nunes went to the building because he needed a secure area to view the information, he told CNN. He said he didn't believe the President nor any of his West Wing team were aware he was there, and the White House said Monday it learned of Nunes' visit through media reports and directed any questions to the congressman.

Nunes defended himself later Monday , telling CNN's "The Situation Room with Wolf Blitzer" that he had to view the classified documents in an Executive Branch location because the intelligence community had not yet provided them to Congress.

"The Congress has not been given this information, these documents, and that's the problem," Nunes said. "This is Executive Branch."

The ranking Democrat on the committee called on Nunes to recuse himself from the Russia investigation.

"But in much the same way that the Attorney General was forced to recuse himself from the Russia investigation after failing to inform the Senate of his meetings with Russian officials, I believe the public cannot have the necessary confidence that matters involving the President's campaign or transition team can be objectively investigated or overseen by the Chairman," Rep. Adam Schiff, a California Democrat, said in a statement.

A former government intelligence official told CNN on Monday that members of Congress, like the general public, must be cleared and escorted into facilities on White House grounds.

"Every non-White House staffer must be cleared in by a current White House staffer," the official said. "So it's just not possible that the White House was unaware or uninvolved."

Trump's press secretary, Sean Spicer, directed all questions Monday about Nunes' source to the chairman.

At a briefing last week, Spicer refused to rule out whether Nunes' source came from the White House but did say during the daily press briefing that "it doesn't really pass the smell test."

"I did not sit in on that briefing," Spicer said. "I'm not -- it just doesn't -- so I don't know why he would brief the speaker and then come down here to brief us on something that we would have briefed him on. It doesn't really seem to make a ton of sense. So I'm not aware of it, but it doesn't really pass the smell test."

Nunes said he was there for additional meetings "to confirm what I already knew" but said he wouldn't comment further so as to not "compromise sources and methods." A spokesman for Nunes said he "met with his source at the White House grounds in order to have proximity to a secure location where he could view the information provided by the source."

A government official said Nunes was seen Tuesday night at the National Security Council offices of the Eisenhower building which, other than the White House Situation Room, is the main area on the complex to view classified information in a secure room.

The official said Nunes arrived and left alone.

Nunes told CNN he had been working on nailing down the surveillance information before Trump's unsubstantiated claim earlier this month that he was wiretapped by President Barack Obama. Last week, Nunes told CNN he was unaware of any evidence to back up Trump's claim.

He told CNN Monday he wanted to "reiterate this has nothing to do with Russia." In a statement, Nunes' spokesman, Jack Langer, said the congressman is "extremely concerned by the possible improper unmasking of names of US citizens."

Two congressional sources said Nunes was with a staff member Tuesday night when he got a message, got out of the car and got into an Uber. Staff did not hear again from him that night.

They next heard from Nunes Wednesday morning, the day he scheduled a news conference before going to the White House. The staff do not know where he went Tuesday night.

Nunes pushed back strongly against an account in The Daily Beast that suggested efforts of subterfuge in his path to his sources that day.

"I was in a cab with staff and we dropped them off before I went to my meeting," he said. "Anything other than that is just false."

'He owes us an explanation'

Nunes' disclosure that Trump's own communications may have been picked up in "incidental" collections by domestic spies -- and decision to speak to the press and White House before informing the Democrats on the Intelligence Committee -- infuriated Democrats and led him to apologize to his colleagues on the panel. Trump has said he feels "somewhat" vindicated by Nunes' findings.

Some Democrats have said Nunes' actions mean he can't conduct an impartial investigation into potential Trump-Russia ties, though House Speaker Paul Ryan said Monday he has "full confidence" Nunes can oversee the probe.

But Rep. Eric Swalwell, a California Democrat who also sits on the Intelligence Committee, accused the White House Monday of "obstructing" the panel's investigation and called for an independent commission to review the matter.

And Rep. Jim Himes, D-Connecticut, told CNN's Kate Bolduan Monday that Nunes' actions were "bizarre" and "loopy," adding that Nunes has told Democrats "nothing" about developments in the investigation.

"Whatever it is he's done, it has been at the White House, it appears to have been in the service of the White House, and so, it is very clear that he owes us an explanation," Himes said on "At This Hour."

This story is being updated.

CORRECTION: An earlier version of this story misstated White House press secretary Sean Spicer's answer Monday to whether Rep. Devin Nunes' source came from the White House. Spicer referred questions about Nunes' source Monday to the House Intelligence Committee chairman.