President Obama, speaking publicly for the first time about his administration's mass collection of phone and Internet data, said Friday that the programs have made a difference in tracking terrorists and are not tantamount to "Big Brother."

Obama addressed the mounting controversy about the programs during remarks in California, ahead of a highly anticipated meeting with Chinese President Xi Jinping. As the administration moves to declassify some of the details about those programs, Obama assured that his administration is trying to strike the right "balance."

"In the abstract, you can complain about Big Brother and how this is a potential program run amok, but when you actually look at the details, then I think we've struck the right balance," Obama said.

Obama acknowledged that the U.S. government is collecting reams of phone records, including phone numbers and the duration of calls, but said this does not include listening to calls or gathering the names of callers.

"Nobody's listening to the content of people's phone calls," Obama said.

As for the collection of Internet data from some of the biggest U.S. tech firms, Obama stressed that it does not apply to U.S. citizens or people living in America.

The administration determined "the modest encroachments on privacy that are involved in getting phone numbers and duration without a name attached and not looking at content, that on net it was worth us doing," he said.

"You can't have 100 percent security and also then have 100 percent privacy and zero inconvenience. We're going to have to make some choices as a society," Obama said.

Obama, though, welcomed a "debate" over that issue.

Several members of Congress are warning they plan to rein in the administration's data collection in light of the revelations.

"This is a dragnet. It's an overreach," Rep. Jim Sensenbrenner, R-Wis., told Fox News.

On top of a Guardian newspaper report that revealed how authorities were collecting phone records from millions, a Washington Post report detailed another program that scours major Internet companies including Google and Facebook for data. A former senior NSA official confirmed to Fox News that the program was started in 2007 by the FBI and NSA and allows them to tap into top U.S. Internet companies to pull audio, video and other data.

Obama said Friday that the intelligence gathering is subject to strict oversight by Congress and a special court, and has been authorized repeatedly since 2006.