I celebrate Judge Richard J Leon's opinion that the government's mass collection of communications metadata is "almost Orwellian", and I decry Judge William H Pauley III's decision that the NSA's collection is both effective and legally perfectly peachy.

But I worry that the judges, as well as many commentators and Edward Snowden himself, may be debating on the wrong plane. I see some danger in arguing the case as a matter of privacy because I fear that could have serious impact on our concept of knowledge, of what is allowed to be known and thus of freedom of speech. Instead, I think this is an argument about authority – not so much what government (or anyone else) is allowed to know but what government, holding unique powers, is allowed to do with what it knows.

Indeed, the Fourth Amendment, which is often called upon in this argument, is explicitly about controlling authority:

The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated, and no Warrants shall issue, but upon probable cause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized.

In the search for a legally protected right to privacy in the United States, begun with Brandeis and Warren in 1890, the Fourth Amendment has been interpreted as affording privacy protection as have the First Amendment (freedom of belief) and the Fifth (freedom against self-incrimination). In each case, though, the right is not so much for something – privacy – as against something – namely, government abuse.

Yet we continue to hold the NSA debate around whether communications metadata is public or private. In the past, such data was presumed to be public because once it was known by a third party, it could no longer be claimed as private. The information on an envelope – metadata to the contents inside: sender and recipient – must be known by a third parties along the way, mail carriers and sorters, to get to its destination. So it is not private. This same theory was applied to the telephone as the phone company has to know who's placing and who's receiving a call to complete it. Thus the government says it can seek such public information without affecting privacy.

Judge Leon argues, with insight, that scale affects the revelatory impact of metadata as we now use phones to do so much more than make calls:

Put simply, people in 2013 have an entirely different relationship with phones than they did 34 years ago.... Records that once would have revealed a few scattered tiles of information about a person now reveal an entire mosaic– a vibrant and constantly updating picture of the person's life.

Yes, but my fear with Leon's argument is that once we we say some amount of data is too much to have, then we will end up debating the line around too much knowledge and that is a line I never want to see drawn. If we start to say that bad things can happen merely if knowledge exists, then too soon we fall into the trap of controlling the extent of knowledge – who may know what and how much they may know and thus who may say what to whom. That is the basis of censorship and ultimately tyranny.

I also fear the impact of Leon's argument on the notion of "publicness". Once knowledge is public, it becomes a public good and the person who put it there does not gain the right to somehow withdraw it because of who ends up holding it or what they may do with it. This is why I object to European Commission Vice-President Viviane Reding's notion of a right to be forgotten – for that gives someone the right to tell others what they may not know. I also object to the idea that there should be a presumption of privacy in public, for that would harm the journalist's – that is to say, anyone's – ability to report on what they witness in public, especially acts by public officials. It could also affect the ability of researchers to collect data and find unforseen connections and correlations.

Think of privacy this way: when I tell you something about myself, that fact is then public to that extent. What happens to it is now out of my hands; it is in yours. Thus, in Public Parts, I defined privacy as an ethic of knowing someone else's information (and whether sharing it further could harm someone) and publicness as an ethic of sharing your own information (and whether doing so could help someone).

When I researched Public Parts, Danah Boyd sat me down and explained how I should understand the gathering versus the use of information:

Privacy isn't just about controlling the access to information but controlling how it's used, how it's interpreted.... If you walk into my office applying for a job, with one quick look I'm going to be able to get a decent sense of your gender, your race, your age.

Antidiscrimination law doesn't forbid her from knowing these bits of information about me. Instead, it forbids her from using them against me in hiring. Of course, she could still deny me the job because of my gray hair. But if she is caught in a pattern of discriminating against applicants on the basis of age, she can be sued.

boyd pointed out an important consequence of restricting use: "If you can't use the information, it makes a lot less sense to try to find ways to access it."

So what we should be restricting – with legislation and open oversight by courts, Congress, the press, and ultimately the people – is the NSA's ability to seek and use information against anyone (citizen or foreigner) without documented suspicion of a crime, due process, and a legal warrant. But don't we already have that: isn't that what the Fourth Amendment prescribes? Well, of course, this is how we end up arguing whether collection of every bit of information my phone provides – whom I talk with and where I go and what I do when – is just collecting data or is the equivalent of searching me or surveilling my every move. Government should not be able to ask for that information unless it has due and just cause to. That surveillance of the innocent is government's overreach of its authority.

But next we end up asking whether that data should be stored anywhere – whether government can decree that phone or internet or credit card companies should hold onto data so government could ask for it. That, I believe, should be governed by a separate set of principles, consumer principles that consider the benefits and risks to me for allowing such data to be held and that give me transparency into what is being done and reasonable control over it. That does not and cannot mean that I can exercise full control over any data to which I'm a party, for data is produced by interactions among parties, each of whom has interests and rights.

We should have this discussion on a level of principles. The best example of that: if our First Class mail carried by the US Postal Service is protected from government search except with a warrant, then all our private communication – by email, direct message, chat, Skype, or any invention to come – should receive similar protection. If metadata at a large scale – phone data – is problematic for government to hold, then shouldn't there be limits on it at a small scale, including the Post Office (which is now photographing and logging every item it handles)? The problem is that these laws and cases were written to a technology – physical mail or POTS -- when they should be written to a principle.

It is also important not to presume that metadata – or Big Data – is bad and dangerous any more than it is right to assume that technology is bad just because it could be misused. I enter into a transaction with Google's Waze allowing it to know where I live and work so I can get the traffic between those points every day. I allow Google to retain my searches (it's easy to use incognito mode instead) because I value more personally relevant search results. I have been arguing that my local newspaper should gather signals about me as Google does so it could give me less noise and more relevance. I understand that Target has to communicate my debit card and pin data to complete a transaction but I expect them to hold that information securely. I also think that my cancer hospital, Sloan-Kettering, should collect data about how many penises, including mine, still function properly after prostate surgery there because that information and associated metadata about surgeons and age and other conditions could be valuable to the patients who follow. Of course, I expect that data to be held anonymously.

Each of these transactions enables the collection and use of data but is governed by sets of principles that take into account the transactors' interests and rights and responsibilities, and those principles should be made public so customers can make decisions based on them. (See Doc Searls' vendor relationship management as an attempt to codify that.)

Government's access to that data must be determined, in turn, by a separate and much more stringent set of laws born of the principles set forth in the Bill of Rights and built with the knowledge that government has the means to use our information against us, in secret. Does the NSA's mass collection, analysis, and use of communications metadata violate the Fourth Amendment? I think it does because it acts as surveillance over innocent citizens, treating all of us as criminals in government's dragnet without probable cause or due process. Or as Jay Rosen puts it: "My liberty is being violated because 'someone has the power to do so should they choose.' Thus: It's not privacy; it's freedom."

Privacy is important. It needs protection. But the primary issue here isn't privacy. Nor is it the existence of any technology or of data. The issue with the NSA in all its activities revealed by Edward Snowden – not just the collection of phone metadata but also the wholesale hoovering of communication on the internet, the creation of backdoors in technology and other efforts to subvert security, the spying on other nations' officials and companies – is government overreach and the absence of oversight. I am less concerned with what government knows about me than what we don't know about government.