By now, most people are aware that the NSA collects massive amounts of information on ordinary Americans. NSA Chief Keith Alexander would tell you that the government must collect the entire haystack to find the needle. But what happens to the rest of the haystack – the information about law-abiding citizens that gets swept up under ever-expanding collection authorities? The answer might surprise you.

In theory, at least, some agencies are supposed to treat the hay differently from the needle. Since the Reagan era, the NSA has been required to be scrupulously careful with information involving Americans. US citizens and others in the United States may not be targeted without a warrant, and "incidentally collected" communications involving Americans may be kept or shared only under specified procedures.

In practice, the picture looks quite different.

Americans' communications are supposed to be destroyed as soon as possible, but they can be kept for up to six years to see if they meet certain criteria, according to recently declassified guidelines (pdf). Metadata about nearly every phone call made within the United States, kept in another NSA storehouse, can be saved for five years. And a recent New York Times report revealed that the NSA keeps a wide range of information about Americans' communications for up to five years in online databases and another ten years "offline for 'historical searches'".

In addition, many other government agencies retain information about innocent Americans, according to a new report from the Brennan Center for Justice. Take the Federal Bureau of Investigation. As its mission transformed after 9/11 from crime-solving to terrorism prevention, the bureau dramatically expanded its legal authority to gather information about Americans with no basis for suspicion. At the same time, few if any additional restrictions were imposed on its powers to keep and share that information.

Today, an FBI agent can open an intrusive investigation with no reason to suspect criminal activity, and any resulting information can be kept for 20-30 years, even if it has no relationship to the investigation. Similarly, the FBI keeps so-called "suspicious activity reports" that are determined to have no relevance to terrorism – but may reflect Americans' constitutionally protected speech or other activities – for 30 years in a widely-accessible database.

Another example is the National Counterterrorism Center, established in the years after 9/11 to serve as a central repository for terrorism-related information. The center issued guidelines last year allowing it to keep and search non-terrorism databases of Americans' information for up to five years, a ten-fold increase over the previous limit.

Why should we care that the government may keep and share information about us? If the government is only looking for terrorists, the vast majority of us surely have "nothing to hide".

But the government's broad sweep for information can land innocent Americans on watchlists from which it is difficult, if not impossible, to extricate themselves. Furthermore, history teaches that the accumulation of personal information about law-abiding citizens carries tremendous potential for abuse – including harassment of minorities, political enemies, and social activists.

There is another major problem: when the haystack gets too big, it is almost impossible to find any needles. Experts across the spectrum have concluded that failures to predict and prevent recent attacks and near-misses – think of the Fort Hood shootings and the 2009 "underwear bomber" – were caused in part by too much information, not too little.

More fundamentally, keeping information about ordinary Americans "just in case" upends the traditional relationship between a democracy and its people. It effectively establishes a presumption that citizens are potentially guilty until proven innocent, and that the government has the right – even the responsibility – to stockpile information that may eventually prove their guilt.

What can be done?

Law enforcement and intelligence agencies should operate under the presumption that they may keep information about Americans only if there is a "reasonable suspicion" of criminal activity. This is a minimal baseline. It would allow the retention of data so long as the government has more than an unsubstantiated hunch. Other steps could include enhanced public reporting from the National Counterterrorism Center, and auditing of all agencies' collection, retention, and use of Americans' information. Although such steps may seem impossible in a period of governmental paralysis, Americans' privacy has proven to be one of the year's few bipartisan issues.

The summer's revelations jump-started a critical public conversation. We must continue it by challenging Congress and administration officials to impose reasonable limitations on law enforcement and intelligence agencies' retention and use of Americans' information. Our national security and civil liberties depend on it.