Getty Images

Facebook lets you control your data -- that's the idea Mark Zuckerberg returned to over and over this week as he testified before US lawmakers.

But some in Congress weren't impressed with that response, including Rep. Ben Lujan, a Democrat from New Mexico. To learn more about what information Facebook collects beyond what users knowingly hand over, Lujan asked Zuckerberg on Wednesday about something called "shadow profiles."

The question hit on an issue that loomed over the hearings this week: Do internet users really know everything that Facebook knows about them?

Zuckerberg demurred, saying he didn't know what a shadow profile is, and to be fair, it's not a term Facebook uses, at least publicly. But privacy advocates use the term to describe something very specific: Facebook amasses information on you that you didn't hand over yourself. That can happen whether or not you're a Facebook user.

At Wednesday's hearing before the House Energy and Commerce Committee, the Facebook CEO confirmed the company collects information on nonusers. "In general, we collect data of people who have not signed up for Facebook for security purposes," he said. And in the past, Facebook has described various forms of data collection that don't involve users directly giving it to the social network.

Facebook didn't immediately respond to a request for additional comment.

That data comes from a range of sources, said Nate Cardozo, senior staff attorney at the Electronic Frontier Foundation. That includes brokers who sell customer information that you gave to other businesses, as well as web browsing data sent to Facebook when you "like" content or make a purchase on a page outside of the social network. It also includes data about you pulled from other Facebook users' contacts lists, no matter how tenuous your connection to them might be.

"Those are the ones we're aware of," Cardozo said.

On Wednesday, Lujan pressed Zuckerberg on what data Facebook collects on people who don't have accounts with the social network. But Cardozo told CNET that most of the data in shadow profiles probably pertains to people with Facebook profiles, "which is, of course, most people."

That's because the company uses the information to show you tailored ads. That means that people who don't use the social network are "not the highest value profile for Facebook," Cardozo said.

Still, Lujan pointed to the challenge faced by people who don't use Facebook but want to see what the social network knows about them.

"It may surprise you that, on Facebook's page, when you go to 'I don't have a Facebook account and would like to request all my personal data stored by Facebook,' it takes you to a form that says go to your Facebook page, and then, on your account settings, you can download your data," Lujan said.

The fact that Facebook has this data isn't new. In 2013, the social network revealed that user data had been exposed by a bug in its system. In the process, it said it had amassed contact information from users and matched it against existing user profiles on the social network.

That explained how the leaked data included information users hadn't directly handed over to Facebook. For example, if you gave the social network access to the contacts in your phone, it could have taken your mom's second email address and added it to the information your mom already gave to Facebook herself. During the time of the data breach, your mom might then have downloaded her information from Facebook, only to find that second email address listed by her name.

The purpose of that data collection was to help Facebook get better at recommending friends for you, the company said.

"When people upload their contact lists or address books to Facebook, we try to match that data with the contact information of other people on Facebook in order to generate friend recommendations," Facebook said in a post explaining how the contact data eventually got leaked.

That raised questions of consent, said Laura Gowans, chief operating officer of privacy-oriented tech company SpiderOak. "The problem is if my friend uses that app and has my contact info, she's consented and I haven't, but Facebook still has my information," she said.

And right now there's a ton of ways for Facebook to get information about you from sources other than yourself.

"It's really, really hard to control what information of your own is getting out there," Gowans said. To keep everything locked down, "you would have to maintain a fake phone number, or never give anybody your phone number or address for any reason."

CNET's Shara Tibken contributed to this report.

Cambridge Analytica: Everything you need to know about Facebook's data mining scandal.

iHate: CNET looks at how intolerance is taking over the internet.