The fresh revelations about ASIS's role left even some defence hardheads shocked. Australia's foremost academic intelligence specialist, the Australian National University's Professor Des Ball, said he found it mortifying that ''we used our highly professional security agency to get a couple of bloody percentage points in our dealings with a struggling country like East Timor. I've got no problems doing it to the Chinese or Japanese, who are doing it back to us … but Timor? That's what bullies do''. Former senior Defence Department official Allan Behm agreed, slamming the spying action against Dili as as ''affronting'' and ''not moral''. The revelations capped off a shocking few weeks for the Australian intelligence community, given recent exposure of 2009 plans by the Australian Signals Directorate to tap the private phones of the Indonesian president, his wife and other members of their inner circle. With Dili and Jakarta in uproar, many Australians might have been wondering what havoc our spy services were going to wreak next.

But red faces have not been confined to Canberra. Its been an annus horribilis for the Western intelligence community as a whole, with continuing disclosures from the innermost vaults of signals intelligence agencies in Britain, the US, Australia, New Zealand and Canada, courtesy of former US intelligence insider Edward Snowden. Snowden, branded a ''traitor'' by Attorney-General George Brandis, has lifted the lid on a surreptitious global communications and data surveillance dragnet operated by the five countries under what is formally called the UKUSA intelligence-sharing pact, known colloquially as the ''Five Eyes'' arrangement. The five eyes operate through the key eavesdropping agencies in each country: the all-powerful National Security Agency in the US, the Government Communications Headquarters (GCHQ) in Britain, New Zealand's Government Communications Security Bureau (GCSB), Canada's Communications Security Establishment and the Australian Signals Directorate or ASD (formerly the Defence Signals Directorate). Decades of intimate intelligence-sharing has left these agencies with a pattern of co-operation that one former senior government insider describes as ''seamless, instinctive, visceral''.

''It has produced a situation where the agencies wouldn't contemplate for a nanosecond not co-operating if one of them asked for something,'' the source says. The slow drip of revelations from the cache of documents, which former NSA contractor Snowden took with him and then leaked to The Guardian, is building an extraordinary picture of the depth, breadth and reach of this global surveillance effort. Just two days ago, The Washington Post (which is partnering The Guardian on some Snowden stories) revealed the NSA was collecting upwards of 5 billion mobile phone records a day, allowing the agency to plot the movements and connections of an enormous number of individuals around the world. The paper quoted an unnamed NSA manager confirming that the agency was getting ''vast volumes'' of location data by tapping into global communications networks. Snowden's cache has unveiled the existence of a veritable alphabet soup of programs - with code names including Prism, Tempora, xKeystroke, Muscular, Pinwale, EgotisticalGiraffe, Stormbrew, Fairview, Oakstar, Mainway, and Nucleon - all aimed at harvesting, storing and analysing as much of the world's electronic communications as can be scooped up.

Among the key programs is Tempora, which allows the NSA and GCHQ, with help from ASD, to tap into the arteries of the global communications system, the undersea fibre optic cables which snake between continents and carry well over three quarters of the word's internet and communications traffic. Then there is Prism, which has sparked particular outrage as it points to extensive NSA penetration of services such as Google, Yahoo, Facebook and YouTube. As one leaked slide prepared by the NSA's Special Source Operations group points out: ''Much of the world's communications flow through the US. Your target's communication could easily be flowing into and through the US.'' Behm says: ''If you and I are on a mobile, neither of us knows quite where the packages of our signals are actually going. It could be going through San Francisco, Canada or somewhere else. The NSA can collect it because it's all up there to be collected, it's just all out there.'' In an Australian context, the documents trickling out so far have raised more questions than they've answered. How far does ASD contribute to the worldwide five eyes data-harvesting effort? How secure from snooping by the intelligence agencies are the US services Australians unthinkingly use on a daily basis, including Google, YouTube, Yahoo, Facebook, Skype, Hotmail and Microsoft?

And even if Australian law purports to offer some measure of privacy protection, how can that be more than a fig leaf when most of these services are headquartered outside Canberra's jurisdiction, and so much of our data is routed offshore? ..... Behm says the legislation governing Australia's intelligence agencies was fashioned at a time when ''everybody thought they knew the boundaries within which those organisations would operate. But since then, there has been an exponential explosion in the amount of data that is available, and in the capacity to store it, monitor it, interrogate it. How do we assess the adequacy of laws [governing the agencies] which were passed more than 10 years ago?'' A salient warning has come from the US founder of Lavabit, Ladar Levison, who shut down his encrypted email service earlier this year because he said he would become ''complicit in crimes against the American people'' if he complied with secret court orders forcing his co-operation with security agencies. ''I would strongly recommend against anyone trusting their private data to a company with physical ties to the United States,'' he warned.

San Francisco's Kurt Opsahl, a senior attorney with the US Electronic Frontier Foundation, describes the latest revelations as the template for a ''surveillance state that is completely global in scope and is designed to use guilt by association: to look at who is in the same location at the same time as others, and then make assumptions based about that information''. Intelligence agency heads in Britain and the US have been at pains to point out that most of the information gathered in mass data-harvesting operations is metadata, key details like time, place, duration and user ID which are automatically recorded when a communication takes place, rather than the contents of the communication itself. NSA director General Keith Alexander has defended ''bulk'' collection as an essential anti-terrorism tool, saying ''you need the haystack to find the needle''. But privacy advocates say metadata can be rich with information about a person's life and associations. University of NSW associate professor and fomer army intelligence analyst Clinton Fernandes says: ''I am able to control what I say on the phone [but] I can't control my metadata.

''Let's say somebody was to ring a rape crisis hotline, followed by an abortion clinic hotline, followed by a counsellor. You wouldn't need to know what the calls were about, you could tell immediately what was going on. So metadata is very revealing in that sense.'' The Snowden documents have so far revealed very little about the precise contribution of Australia to mass data-gathering and monitoring operations. One snippet, apparently jotted down by note-takers at a 2008 meeting at GCHQ in Britain, suggested the then Defence Signals Directorate could ''share bulk, unselected, unminimised metadata as long as there is no intent to target an Australian national''. It added that ''unintentional collection is not viewed as a significant issue''. Although some have seen this a smoking gun, the Defence Department (of which ASD is a part) issued emphatic denials this week. ''ASD is a foreign signals intelligence organisation which targets foreign communications,'' a Defence spokesman said. ''ASD does not undertake the mass collection of domestic metadata. ASD can only produce intelligence, including metadata, about Australian persons in a limited range of circumstances subject to a ministerial authorisation first being obtained.'' Ball and Behm find these assurances convincing. Ball is adamant the agency is '' very much involved in this global activity but not when it comes to picking up Australians''.

But others argue there is such seamlessness to the five eyes' intelligence gathering that nothing stops the other partners combing through Australian traffic which they pick up, and feeding the results back to our agencies for ''target development''. The NSA and the GCHQ each used the other to circumvent domestic restrictions on intelligence collection. There is little transparency in Australia, either, about how long domestic telecommunications agencies and internet service providers are storing communications records to meet legislative requirements that they assist police and intelligence agencies. (Demands for metadata do not require ministerial warrant.) ''Australians would be gobsmacked if they understood how much data telecommunications companies had on them,'' says one observer with some knowledge of Telstra operations. ''Telstra and some of the other major telcos are virtual honey pots of information because of this ceaseless river of data flowing into their information banks. We just don't know how how much of that the agencies have access to.'' Fairfax revealed on Friday that Telstra had installed ''vacuum cleaner''-like technology supplied by US firm Gigamon to help it sift through telephone calls, texts, social media and internet metadata, and that ASD was also a Gigamon customer. Telstra denied using ''any traffic monitoring system to conduct mass surveillance on behalf of Australian national security agencies.'' But all the telcos have to comply with secret requests from the police and intelligence agencies.

In Senate estimates committee hearings in Canberra last month, ASIO director-general David Irvine defended his organisation's ability to keep the cloak of confidentiality around metadata collection. ''We are very keen to ensure that … is kept secret so that it is not possible to determine who and what we are targeting,'' he said. In Britain and the US, some leading politicians are starting to question the vast data-gathering operations unmasked by Snowden. Britain's deputy prime minister Nick Clegg recently declared that ''with each passing day there is a stronger and stronger case'' to question ''the proportionality of intelligence-gathering today and the accountability of the services''. In the US, Barack Obama has ordered a snap review of NSA powers. But in Australia, the political response, with the exception of Greens senator Scott Ludlam and independent senator Nick Xenophon, has been anaemic.

Apart from relevant ministers, there is meant to be a special parliamentary committee which keeps watch on the agencies - the Parliamentary Joint Committee on Intelligence and Security - but it has yet to be reconstituted since the September election. And its powers of scrutiny are limited. There is also an Inspector-General of the Security Services (IGIS), Dr Vivienne Thom, but critics claim her office is under-resourced and too much part of the intelligence ''club''. The president of Civil Liberties Australia, Dr Kristine Klugman, recently wrote to Dr Thom, urging her to investigate the impact of the NSA's Prism program on Australians. The IGIS replied that she lacked power to ''look at the activities … of foreign agencies'' and in any case would not comment on ''operational matters of the intelligence community''. Ball says the IGIS office is ''minute, overseeing very secret, very compartmentalised and technically complex [ASD] operations. At the moment, I suspect it's beyond them''. He and Behm, strong supporters of the core work of the espionage agencies (which included tracking down some of the Bali bombers), believe there has been ''mission drift'' and that public confidence needs to be restored.

Loading Says Behm: ''I think given both the WikiLeaks and Snowden information, it's high time that parliaments, in the United Kingdom and Australia, had a properly constructed inquiry as to whether existing legislation meets the needs that citizens have for the protection of their fundamental democratic rights which includes privacy.'' with Ben Grubb