More than five million people have national security clearance in the United States, according to a new report – that’s more than the entire population of Norway. The startling number is leaving many people wondering who has access to our private records.

On America’s News HQ, Judge Andrew Napolitano explained that there are four different categories of secrets. The president and the people he has designated are in the highest category.

Citing the NSA spying scandal, Judge Napolitano believes there are too many people with access to Americans’ private information. As a result, he said it creates a society of “us against them.”

“The problem is not that we put secrets into the hands of people who are entitled to them,” the judge said. “The problem is that we willy-nilly characterize things as secrets that shouldn’t be.”

He said the number of people with access to citizens’ information should be “radically reduced.”

President Obama announced plans to ask Congress to end the bulk collection of data by the NSA. However, Judge Napolitano said the proposal is only cosmetic.

“I don’t think the president is going to change anything. Instead of having the NSA keep the records, he’s going to have the telephone and the computer server companies keep them.”

The judge warned, “[The NSA] will gather more secrets and … they will all know more about us until our children and our grandchildren won’t even know what privacy means.”