A culture of secrecy, unaccountability and outsourcing created the conditions for the Obama administration's mass internet surveillance programs.

You have to wonder whether, when Xi Jinping, newly-installed President of one of the world’s most brutal dictatorships, met Barack Obama in California on the weekend, he asked his American counterpart if he had any good tips for mass surveillance and breaking into the networks of the world’s biggest internet services companies.

Only a few hours earlier, Obama had admitted that there was indeed, as per the revelations of Edward Snowden, mass internet surveillance programs run by the National Security Agency. The likes of Microsoft, Google and Apple had vociferously denied the NSA had any access to their servers. They were either lying, or their CEOs had been kept ignorant by their lawyers because of the remarkable secrecy demands of the Foreign Intelligence Surveillance Act. Or the NSA had illegally broken into their systems. It rapidly became clear that it wasn’t the last, as the extent of Silicon Valley cooperation with the NSA was revealed.

All of which rather put a dampener not merely on Obama’s plan to confront Xi Jinping over cybersecurity but on the whole “China is the world’s biggest hacker” line that governments and the mainstream media have insisted on running.

“We hack everyone everywhere … we are in almost every country in the world,” one intelligence source admitted to Glenn Greenwald, who broke the Snowden revelations. Former NSA and CIA director Michael Hayden boasts that the US is the best in the world at online attacks.

Edward Snowden, the whistleblower who revealed the population-wide extent of NSA surveillance, worked for Booz Allen Hamilton inside the NSA. How could he work for a private company inside a secretive intelligence agency like the NSA? Because for nearly two decades the US government has been outsourcing a vast array of national security functions. Why? Notionally it allows the US government to tap into private-sector expertise in areas of online communications where it struggled to keep up. But it’s also because it makes vast amounts of money for private defence and security companies. The Pentagon cybersecurity budget over the next five years is likely to rise to US$23 billion. Up to 70% of it will go to private contractors like Booz. And those companies are where most senior defence and intelligence officials will eventually work.

Take, for example, John M (Mike, apparently) McConnell. He was a former head of the NSA who was George W. Bush’s director of national intelligence (DNI). But before that, the one-time Rear-Admiral had been senior vice president of Booz. And after McConnell left the DNI role at the start of the Obama administration, he returned there. He’s still there now.

Thanks for signing up We look forward to seeing you bright and early with your need-to-know talking points and tidbits for the day ahead. Get Crikey FREE to your inbox every weekday morning with the Crikey Worm. Please enter your email address Sign up

McConnell is the cyberhysteric from central casting. “We have had our 9/11 warning,” he told Congress last December. “Are we going to wait for the cyber equivalent of the collapse of the World Trade Centers?” But not merely does he warn of a digital 9/11, and of course warn of the threat posed by China, but he calls for more US government spending on cybersecurity.

“Securing cyberspace will require a more robust commitment in terms of leadership, policies, legislation and resources than has been evident in the past,” he said in February 2010. Shortly afterward, Booz picked up a lazy $34 million from the US government in cybersecurity contracts.

Fans of outsourcing, such as noted supporter of IRA terrorism Peter King, were quick to suggest that private staff employees were no more likely to be a security risk than government employees. But outsourcing expands the intelligence community (an absurd term, suggesting a folksy group of friendly, smiling cut-throats and informants, but we’ll stick with it), growing the base of firms that can lobby for government funding and also increasing the risk of a leak simply through sheer maths.

“The result is a multi-billion dollar industry employing hundreds … of companies spying on Americans and non-Americans alike, with no public oversight or accountability of any kind”

That’s what happened with Snowden, the man behind the Verizon and PRISM leaks, who decided that he became desperate to see the “federation of secret law, unequal pardon and irresistible executive powers that rule the world that I love are revealed even for an instant”. A key motivation for Snowden appears to have been the secrecy with which the US government had set out “to destroy privacy, internet freedom and basic liberties for people around the world with this massive surveillance machine”.

The leaks, then, suggest that the information asymmetry that governments crave can be counter-productive. An important feature of the response of governments and companies to the challenges of online communication is a desire to maximise the utility of the internet for surveillance — knowing all about citizens — while minimising how much citizens know about their governments.

Compared to Australia and, perhaps, the UK, the US government is significantly more asymmetrical; the de facto US data retention regime exposed last week, for instance, operated entirely in secret, whereas the UK and Australian governments have publicly debated data retention.

Indeed, the US government’s obsession with information asymmetry reached its ultimate extent last week when, in response to the leak about Verizon, McConnell’s successor as DNI, James R. Clapper, complained:

“The unauthorized disclosure of a top secret U.S. court document threatens potentially long-lasting and irreversible harm to our ability to identify and respond to the many threats facing our nation.”

Thus, not merely do governments not want you to know about them, but governments claim that “long-lasting and irreversible harm” could result not so much from you knowing about your government, but from knowing what your government knows about you.

In the US, the asymmetry extends not merely to the extent of spying, but the legal framework within which that spying is conducted, due to a combination of four things — a secret court called the Foreign Intelligence Surveillance Court, which adjudicates agency requests under FISA; the Obama administration’s reliance on secret legal opinions it uses to determine the extent of its powers; a willingness of Congress and congressional committees to keep secret what oversight it does have of spying; and a willingness on the part of intelligence authorities to lie to Congress. It’s only three months since DNI James Clapper told Congress the NSA did not collect information on “millions or hundreds of millions of Americans”.

This lack of oversight and accountability has a direct impact on the culture of spying. The NSA has long targeted Greenpeace as a security threat, for example. In 2008, a whistleblower revealed that NSA spies had eavesdropped on phone calls from American military personnel, journalists and NGO staff in the Middle East. This wasn’t in pursuit of terrorism, but simply to listen to intimate discussions between partners for titillation.

When government agencies have access to information and no fear of public accountability, mission creep, laziness and stupidity creep into its usage. So does politicisation. The Obama administration has launched a campaign on whistleblowers and investigative journalists using its secret powers of information-gathering, claiming it is about national security but in reality — as was demonstrated by the pursuit of Associated Press journalists’ phone records — it is about discouraging embarrassing media coverage.

The result is a multi-billion dollar industry employing hundreds, if not thousands, of companies spying on Americans and non-Americans alike, with no public oversight or accountability of any kind, using mass surveillance for political and economic goals unconnected with security, or simply to listen to soldiers’ phone s-x.

The only way to keep this vast corporate-state hybrid even remotely accountable, to start reversing information asymmetry, is via leaks.