The long-running debate over the tradeoffs the United States should make between national security and civil liberties flared up spectacularly last summer, when Edward Snowden, a National Security Agency contractor, handed journalists a huge trove of heavily classified documents that exposed, in excruciating detail, electronic surveillance programs and other operations carried out by the NSA. Americans suddenly learned that in recent years, the NSA had been acquiring the phone and Internet communications of hundreds of thousands of U.S. citizens, as well as collecting massive volumes of bulk telephone records known as “metadata” -- phone numbers and the time and length of calls. Along with the rest of the world, Americans found out that the NSA had broken common forms of online encryption, tapped the phones of various foreign heads of state, and monitored global communications far more aggressively than was previously understood.

Howls of outrage erupted. Brazilian President Dilma Rousseff, who learned from the Snowden leaks that the NSA had been monitoring her personal conversations, described the NSA’s activities as a “violation of human rights and civil liberties,” decrying the “disrespect to national sovereignty.” In the United States, both ends of the political spectrum denounced the NSA’s activities. Rand Paul, a Republican senator from Kentucky, called them “an all-out assault on the Constitution,” and the former Democratic vice president Al Gore said they were “obscenely outrageous.”

Proposals for reform are now legion. Soon after the leaks, President Barack Obama appointed an independent group of experts to examine the issue. The group’s report, published last December, recommended more than 40 far-reaching reforms, including ending the government’s bulk collection of telephone metadata and restricting surveillance on foreign leaders. The panel suggested that telephone providers or a private third party, not the government, should hold the metadata and give officials access to it only when ordered to do so by the secret Foreign Intelligence Surveillance Court. The panel also recommended requiring the FBI to obtain judicial approval before issuing a “national security letter,” a form of administrative subpoena the government uses to obtain phone numbers, e-mail addresses, and financial transaction records. Congress is also mulling action. Last October, Patrick Leahy (D-Vt.), chair of the Senate Judiciary Committee, and James Sensenbrenner (R-Wis.), former chair of the House Judiciary Committee, proposed a major rollback of the NSA’s programs. At the same time, the Senate Intelligence Committee put forward a modest bill that tinkered with, but largely validated, the current legal status quo.

Obama responded to the public outrage and various calls for reform with a major speech and a presidential policy directive in January. Obama defended the NSA, emphasizing the necessity of intelligence and noting that nothing he had learned “indicated that our intelligence community has sought to violate the law or is cavalier about the civil liberties of their fellow citizens.” Yet Obama also warned that given the NSA’s power, the U.S. government has a “special obligation” to scrutinize the agency’s activities. He acknowledged that non-U.S. citizens overseas have privacy interests that the United States must respect. He also restricted the NSA to obtaining specific records only with an order from the Foreign Intelligence Surveillance Court and only on targets that are two steps removed from the phone number of a suspected terrorist, rather than the previous three steps. Obama also declared a “transition” to end the government’s collection of bulk telephone metadata; two months later, as this article went to press, the administration was reportedly proposing to change the system so that phone companies would store the metadata, rather than the government.

Although bold on principles, Obama’s plans for reform have been vague on some important details. In January, he said that the United States would still spy on non-U.S. citizens, including foreign leaders, if “there is a compelling national security purpose” -- quite a caveat. And he called for working with Congress -- not exactly a sure-fire source of action -- on national security letters, but he did not mention requiring the FBI to obtain judicial consent before demanding records.

The proposed reforms and Obama’s less than full embrace of them reflect a fundamental clash when it comes to what the American public demands of its intelligence community. The real problem that Snowden’s revelations brought to light was not a government agency run amok: the NSA never meaningfully exceeded the writ given to it by the White House, Congress, and the courts, at least not intentionally. Rather, those revelations highlighted a basic conflict between two things that U.S. citizens and their government demand from their intelligence agencies: a high, if not perfect, level of security, on the one hand, and strict privacy protections, accountability, and transparency, on the other. Those imperatives were never easy to reconcile and are even harder to resolve today. Indeed, Snowden’s revelations demonstrated how the implicit bargain that has governed the U.S. intelligence community since the 1970s has broken down.

For four decades, U.S. intelligence work was predicated on a compromise. Covert spying was allowed -- including, at times, against U.S. citizens -- so long as it was subjected to formal, albeit secret, oversight and a sharp distinction was maintained between domestic and foreign targets. Today, however, thanks to both technological developments and the NSA’s increased role in counterterrorism after 9/11, that boundary has become hard to uphold. Meanwhile, the U.S. government’s decreasing ability to keep its own secrets has exposed the flaws in the intelligence compromise.

But that does not mean that the U.S. government should abandon its quest for good intelligence. As communications technologies spread and the overall volume of communication increases, the NSA’s role is growing even as the political space the agency enjoys has shrunk. The question really underlying all the fevered talk of reform is whether the NSA can win back the public’s trust, or at least its acquiescence.

In recent months, a new consensus on intelligence gathering has begun to emerge among a wide swath of the U.S. political establishment, although it excludes critics on the civil liberties left and the libertarian right. The NSA should retain many of its powerful capabilities, but it needs to change the way it thinks about its interactions with the American people and become more open about its operations. While still keeping their collection methods secret, officials have begun -- and need to continue -- to publicly disclose far more about the categories of people the NSA targets for surveillance and how the agency collects and uses information. Officials have also started to accept more public oversight of such activities and the reality that many of the NSA’s secrets about the information it targets, the technology it penetrates, and the rules that govern its activities will inevitably be revealed.

As a result, the NSA will have to determine whether a new program’s benefits really outweigh the potential costs of exposure. Such decisions will come at a price. The new limits on NSA surveillance will at times leave the U.S. government less informed about threats and opportunities. So U.S. policymakers and citizens alike will have to consider how much security and diplomatic advantage they are willing to forgo in exchange for greater restraint from the intelligence community.

THE GREAT COMPROMISE?

The NSA claims that its activities have helped prevent numerous terrorist attacks at home and abroad since 9/11. Such claims are difficult to verify without access to classified data. More important, they rely on an inappropriate measure of success. The agency’s true remit goes beyond just stopping attacks: the NSA seeks to identify terrorists, understand their organizations, and anticipate and disrupt their activities. On that broader set of tasks, the agency has accomplished a great deal in recent years. But however important, the NSA’s data collection is rarely the only factor in effective counterterrorism. Such operations are the result of coordination and cooperation among many different intelligence organizations.

Additionally, the recent debate over the NSA has focused too narrowly on counterterrorism. That attention is understandable: U.S. government officials know that the easiest way to defend the agency after Snowden’s disclosures is to invoke its role in preventing terrorist attacks. Moreover, the collection of bulk telephone metadata -- the most controversial program Snowden revealed -- happens to be a counterterrorism program. But the NSA does vital day-to-day work in the realms of diplomacy and cybersecurity as well. By accessing the communications of foreign leaders and officials, particularly of U.S. adversaries, the NSA provides U.S. policymakers with insights into when a state might go to war, break a treaty, or otherwise make a dramatic (or subtle) policy shift.

NSA activities allow U.S. officials to negotiate more effectively by tipping them off to the positions of foreign officials. That advantage applies even to relationships with allies, with whom the United States maintains extensive intelligence-sharing arrangements. The intelligence gathered by the NSA can confirm the accuracy of information that allies voluntarily share with Washington. Even friendly states sometimes choose to shade the facts or share partial information with Washington in an attempt to avoid embarrassment or shape U.S. policy.

In order to accomplish its missions, the NSA has built up a vast array of collection capabilities -- too vast, say the agency’s many critics at home and abroad. Americans do have good historical reasons to be suspicious. In the 1960s and 1970s, the NSA, along with other U.S. intelligence agencies, conducted abusive surveillance of journalists; members of Congress; Martin Luther King, Jr., and other civil rights leaders; and prominent opponents of the Vietnam War, such as Muhammad Ali and Benjamin Spock. After the Watergate scandal, journalists and congressional investigators eventually exposed such snooping, which led to widespread distrust of government surveillance and secrecy. (Some of the names and details of specific targets, however, were not disclosed until last year, in declassified NSA documents.)

To guard against future abuses while also preserving the confidentiality that intelligence agencies require, in the late 1970s, Congress devised a series of oversight committees and other mechanisms that relied on two overarching concessions. First, the new rules granted legislators and judges more oversight over the intelligence agencies but required nearly all their reviews to take place in secret. Second, the 1978 Foreign Intelligence Surveillance Act (FISA) allowed the FBI (the NSA was not permitted to operate domestically) to target the communications of people inside the United States, including U.S. citizens, but required it to obtain approval for doing so from the secret Foreign Intelligence Surveillance Court, staffed by federal judges appointed by the chief justice of the United States. Other rules required the NSA to discard U.S. citizens’ communications inadvertently swept up by dragnets aimed at overseas targets, unless the agency concluded that the data had foreign intelligence value.

Beginning around the same time, the White House and the Department of Justice also increased their oversight of the intelligence community. The net result of all these changes was a system in which the NSA could use its vast powers only in certain circumstances and only under the supervision of a lot of minders. Throughout the 1980s and 1990s, this system seemed to work well. The agency was careful not to target U.S. citizens illegally and avoided using even its limited powers to their fullest extent. As a consequence, the NSA mostly stayed out of major controversies and behind the scenes, its operations at once robust but very much subject to the law.

THREAT ASSESSMENT

In recent years, however, two major changes have deeply upset the status quo, empowering the NSA in the short term but undermining its longer-term support from and legitimacy with the American public. The first change was the profound shift in national security priorities provoked by the 9/11 terrorist attacks. The failure to stop the attacks led to criticism in the media and Congress that U.S. intelligence agencies had been too passive in the face of the growing jihadist threat. Critics charged that the NSA, chastened by the revelations of its abuses in the 1970s, had interpreted its powers too conservatively and had too often hesitated to collect information that might have involved U.S. citizens -- even when those Americans were in contact with suspected terrorists overseas.

In response to such pressure, legislators and officials inside and outside the NSA pushed for more aggressive surveillance measures as counterterrorism, long just one part of the agency’s portfolio, became its priority. President George W. Bush’s warrantless wiretapping program, initiated shortly after 9/11, circumvented FISA procedures and safeguards altogether and, when revealed, led to significant changes in the law itself -- changes that confirmed the agency’s authority to conduct surveillance without individualized warrants on non-U.S. citizens or residents overseas whose communications the NSA collected when they passed through the United States. (The changes made clear, though, that the surveillance had to be limited to those communications and that the NSA still had to have individualized warrants when targeting U.S. citizens or residents abroad.) The post-9/11 quest for more and better intelligence also led to a huge increase in the NSA’s budget, which, according to The Washington Post, totaled almost $11 billion last year, up by over 50 percent since 2004.

The second change was more akin to a tectonic shift: advances in technology began permanently blurring the distinctions between domestic and foreign surveillance and between U.S. citizens and foreign nationals. The Internet and the spread of mobile and wireless devices have vastly increased the extent of international communications by people in the United States, who now frequently interact with people of other nationalities over the Internet.

These technological changes have made it much more difficult to separate domestic and foreign communications. In the 1970s, foreign communications involved large phone circuits and satellite feeds that were largely distinct from domestic communications infrastructure. Today, the data streams have become hopelessly entangled. Messages between Internet users in the United States sometimes travel to and from sites and facilities in Europe or Asia, even if the final destination is an office next door. Foreign communications often go through the United States, a byproduct of the centrality of U.S. companies and infrastructure to the Internet’s technical architecture. When the NSA taps online communications from foreign nationals abroad, it often ends up capturing messages sent between U.S. citizens in the United States. If the agency excluded U.S. sites from its coverage, it might miss out on foreign communications that are routed through or stored on U.S. servers.

These changes have created major technical challenges for the agency -- and a huge boon. The NSA can access people’s Facebook messages, Skype feeds, financial documents, e-mails, and stored computer documents, allowing it to learn exponentially more about a target than it could have in the pre-Internet age. These days, in fact, the NSA’s biggest technical problems involve not collection but analysis. The NSA estimates that it “touches” (without specifying what that means) a cache of information from the Internet equivalent to 580 million file cabinets of documents every single day. The agency can only hope to analyze a tiny fraction of this enormous haul.

These changes also test the limits of the 1970s compromise. Thanks to Snowden’s leaks, a significant portion of the American public now doubts that the NSA truly focuses on overseas communications and has little faith in the government’s oversight mechanisms. The NSA, of course, struggles to keep its own secrets under wraps. The agency can rightly blame leakers for this problem. But given the number of people who now have access to NSA documents (more than a million people have “top secret” clearance), such leaks seem almost inevitable.

The agency has thus begun working even harder to keep its secrets. In an interview with one of us last December, Lonny Anderson, the NSA’s chief technology officer, said that the agency has begun to more closely monitor its employees’ use of agency resources, limit the amount of data that their terminals can access, and centralize its data storage in its internal cloud. Such changes will make the NSA less agile, and at the end of the day, Anderson conceded, “You can never stop someone who’s determined, who . . . has got all the right badges, [and who has] some technical skills.”

Indeed, Washington should now expect that its allies and adversaries, not to mention the general public, will occasionally gain access to at least the general patterns of U.S. intelligence collection. At times, they will even gain access to the specifics. The impact of such revelations will vary. U.S. enemies assume that Washington listens to them, so they are unlikely to be shocked by revelations. But U.S. allies claim to have different expectations -- or at least they did until Snowden’s revelations angered foreign leaders such as Rousseff and Germany’s Angela Merkel, who found out that their personal communications had been intercepted.

SPY GAME

Although some of the foreign outrage was manufactured, the Snowden revelations have hurt the United States’ relations with its allies in two vital ways. First, they surprised and angered publics in allied countries, forcing leaders such as Merkel and Rousseff to respond to -- or exploit -- that anger. Second, the United States has now lost some of the moral high ground it had occupied in debates over cybersecurity and Internet governance. After years of protesting Chinese cyber-intrusions into U.S. systems, Washington now looks hypocritical. In fact, the Snowden documents suggest not just that the United States, like China, engages in cyberspying but that the United States is really good at it.

The revelations have also threatened U.S. technology firms, which many critics now suspect cooperate voluntarily with the NSA and thus essentially operate as an arm of the U.S. government. A presidential policy directive that accompanied Obama’s speech in January acknowledged the risks that such perceptions pose to the country’s “commercial, economic, and financial interests, including a potential loss of international trust in U.S. firms” and “the credibility of our commitment to an open, interoperable, and secure global Internet.” Yet Obama offered technology companies mostly rhetoric, rejecting the review panel’s call for the NSA to stop undermining encryption standards. Major U.S. Internet companies have begun openly contemplating the adoption of more sophisticated encryption methods.

Meanwhile, foreign countries have toyed with the idea of requiring Internet companies to provide local data-storage services to their citizens. Some foreign governments and companies may turn to domestic firms for their technological needs; such firms will be sure to emphasize that their U.S. competitors will not keep foreign data secure. But these efforts may ironically make the NSA’s job easier, since the agency is less constrained by laws or oversight in accessing data stored abroad.

For their part, NSA officials are deeply concerned about the impact of the revelations on U.S. companies. Anne Neuberger, who acts as the agency’s top liaison to the domestic private sector, said in an interview with one us last December that they “feel a sense of responsibility to look at” the damage to Silicon Valley’s reputation after Snowden’s leaks. The NSA simply cannot function without industry cooperation. But as Obama’s speech showed, the administration does not have that much to offer the technology industry in the way of new restraints without seriously inhibiting the NSA’s data collection. The most important gesture to the industry in Obama’s speech was the repeated reference to respecting the privacy of non-Americans, which was designed to reassure overseas individuals about using U.S. software and Internet services.

U.S. intelligence officials shoulder some of the blame for the lack of public confidence in the NSA, since they have not always been completely honest in their public statements. James Clapper, the director of national intelligence, told the Senate in March 2013 that the NSA does “not wittingly” collect data on U.S. citizens. But less than three months later, the massive metadata collection program was revealed, leaving Clapper to lamely claim that his original statement had been the “least untruthful” one he could give at the time. Journalists have also done their part to diminish the public’s trust, sometimes publishing misleading claims, as when the Norwegian daily Dagbladet reported last November that the NSA had collected Norwegian phone conversations -- only to have Norway’s intelligence service turn around and disclose that it had done the spying itself.

THE CHAMBER OF SECRETS

Whatever the cause, the fact is that the NSA, and by extension the Obama administration, now faces a big public relations problem. One response would be to further increase secrecy in the hopes of preventing more embarrassing leaks. Another would be to accept that much more information about U.S. intelligence work is likely to become public in the future and thus to conduct that work more visibly. Washington’s best approach, however -- and the direction the government seems to be stumbling toward -- would be to combine these responses by making hardheaded judgments about what secrets the agency really needs to keep and working even more rigorously to protect them but also adopting a much more open posture toward less sensitive forms of intelligence collection.

Above all, any scaling back of secrecy must be well planned and should proceed according to a coherent theory of how civil liberties can best coexist with surveillance and how transparency can coexist with espionage. That does not seem to be happening now: the intelligence community is currently rolling back surveillance programs and disclosing thousands of pages of classified documents (including dozens of Foreign Intelligence Surveillance Court opinions and orders) that offer information on the telephone metadata collection program and on the targeting of foreigners, all without a clear strategy -- or, at least, without a strategy that has been explained in public. The process seems mostly reactive, an impulsive response to bad press and to blowback from individual foreign governments.

As it contemplates more disclosures and reforms, the U.S. government needs to answer some basic questions about what kinds of surveillance its intelligence agencies currently refrain from and what kinds of surveillance they should conduct more openly. It is not clear how intelligence officials should react if new restraints prove harmful to U.S. security. Nor is it clear whether U.S. officials and the American public would be willing to accept the results of less effective surveillance programs in the name of greater transparency. Obama has begun answering these questions. But some of his answers, such as those related to the technology industry and the privacy of non-U.S. citizens, have left too many details undetermined.

To set a clearer agenda for reform, the NSA should begin by dividing its activities into three broad categories. First, the agency should identify what it really must keep secret. In truth, only a fraction of the NSA’s current activities -- penetrating new technologies, for example, or monitoring supposedly secret systems of U.S. rivals, such as China -- are so sensitive that the mere revelation of their existence would damage U.S. interests.

The NSA needs to work harder to keep those programs hidden by granting far fewer people access to them. Anderson said that stricter controls on access are already in the works, with a system to tag each piece of data that the NSA collects and each user. Data and user tags could then be matched depending on the user’s privileges. Keeping access to the most important secrets limited to a smaller circle of confidants would make it more likely that they stayed secret.

But a push for more secrecy will provoke new fears of future abuses. Keeping fewer people in the loop would also increase the risk that important dots could go unconnected. Anderson acknowledged this risk, saying the agency is currently erring on the side of data security at the expense of effectiveness. There is no way to resolve that dilemma: to preserve secrecy, the NSA will have to forgo the benefit of having lots of eyes on a problem. But this tradeoff is sometimes worthwhile, since it ensures that the most important programs are privy to only a select group of analysts.

When it comes to the agency’s less sensitive work that has not yet been exposed, the NSA should be prepared to abandon it if the benefits do not outweigh the costs of disclosure. Some spying on allies, for example, should be reconsidered, as Obama has already committed to doing. The practice in itself is not wrong, and it often yields valuable findings. But just as often, the benefits are not worth the price.

Third, the NSA must lift the veils over certain programs it means to continue. Because of Snowden’s leaks and subsequent disclosures and declassifications, the metadata collection program, for example, is not a secret -- and so even if some version of it continues, it makes sense to err on the side of openness going forward. More generally, the NSA should disclose more information to the public about the scope and scale of politically sensitive surveillance, where possible, and even more specifics to Congress. Bringing in civil liberties groups to discuss the parameters of some programs involving the surveillance of U.S. citizens would also help. The NSA is not likely to convince such groups to take the agency’s side, but it could still explain to them its procedures for minimizing intrusions.

Becoming more open will require a shift in the institutional culture of the NSA and in the intelligence community more generally. But that shift is already taking place. In 2012, the NSA’s then deputy director, John Inglis, quipped that the agency is “probably the biggest employer of introverts” in the federal government. But over the past few months, the country’s most powerful introverts have begun speaking out publicly to an unprecedented degree. Last December, senior NSA officials even agreed to participate in a lengthy series of podcasts with one of us on the future direction of the agency.

Ultimately, increasing the transparency of the NSA and boosting oversight of its activities will have serious operational consequences. Those changes may at times slow down surveillance or make the agency more hesitant to acquire data that, in hindsight, would have been useful for counterterrorism or other essential operations. But conducting intelligence in public, at least to a certain degree, will help preempt scandals and allow the NSA to educate policymakers and journalists about what it does and why.

Despite Snowden’s leaks, much of the public still misunderstands how the NSA works and what it does. In the past, the agency has welcomed this ignorance, since it helped the government keep its secrets secure. But now that the cat is out of the bag, the NSA, mindful of the value of public trust, needs to recalibrate its operations in order to increase public understanding of how it works. The necessary reforms will, to one degree or another, require Americans to take on more risk -- a decision that will lead to political criticism should another terrorist attack occur on U.S. soil. If done well, however, the reforms will also make the agency more sensitive to public concerns while preserving its necessary core capabilities.