Based on bipartisan laws passed in March 2015, Australian government agencies were given access to the data of millions of citizens without a warrant. Last week, this regime came into effect.

It has been implemented despite ongoing confusions about costs, what metadata actually is, and whether the telecommunications sector is ready to retain and encrypt this data.

At the heart of justifications for the data retention laws is the claim that these will protect us from terrorism. For advocates, mandatory data retention is integral in preventing a 9/11-type attack. But long-standing questions remain as to whether such programs are actually an effective counter-terrorism tool.

In the wake of National Security Agency (NSA) whistleblower Edward Snowden’s revelations, it can be argued that mass metadata surveillance has proven to have little to no unique value in thwarting terrorism. And while the US and others are being forced to revise and restrict their surveillance laws, Australia is moving unreservedly in the opposite direction.

The value and utility of the NSA’s metadata retention programs – which formed the template for Australia’s metadata regime – have too often been exaggerated. Critics, including members of US Congress, have amassed an array of concerns. Among these are potential abuses, a lack of transparency and constitutionality, as well as privacy encroachments by spies and security services.

The utility of mass surveillance in the US

US President Barack Obama correctly pointed out in 2013 that metadata can be used to expose criminality. At the same time he urged:

It’s important to recognise that you can’t have 100% security and then have 100% privacy and zero inconvenience.

However, warrantless mass data collection has not proven to be a leading component in mitigating the terror threat since 9/11. There is little evidence that bulk electronic collection, based on Section 215 of the USA PATRIOT Act, has significantly enhanced national security efforts.

Attempting to provide some public foothold for a cogent cost-benefit discussion, Obama and the NSA have regularly tried rationalising the existence of controversial data collection programs. In June 2013, Obama declared:

We know of at least 50 threats that have been averted because of this information.

This “50 threats” assertion – made by Obama and intelligence officials in pointing towards direct national security benefits – is ambiguous, speculative and even deceptive.

In December 2013, in Klayman v Obama, judge Richard J. Leon contradicted the principal validation of data collection: that it is indispensable for preventing looming terrorist attacks. He said:

I have serious doubts about the efficacy of the metadata collection program as a means of conducting time-sensitive investigations in cases involving imminent threats of terrorism.

The exact figures paired to the disruption of terror “plots” used by Obama, and echoed by the NSA and its advocates, remain extremely debatable. Senators Ron Wyden and Mark Udall said they had not seen:

… any evidence showing that the NSA’s dragnet collection of Americans’ phone records has produced any uniquely valuable intelligence.

Similarly, despite the ostensible post-9/11 imperative for NSA surveillance, such programs often appear to be across-the-board “fishing expeditions”. They have proven to be a principally inconsequential tool in foiling domestic terror plots by homegrown extremists.

The case of Najibullah Zazi, who planned a New York City subway attack in 2009, highlights the minor role of metadata collection in the detection of such plots. It was found the the government’s collection of phone records from millions of Americans had little to do with its detection. Instead, it was later revealed that traditional police work and court-approved wiretaps were key in uncovering Zazi to authorities.

In January 2014, an extensive survey of unclassified cases by the non-partisan New America Foundation reinforced this. It showed that the mass collection of phone metadata had no “discernible impact” in thwarting acts of terrorism, with negligible national security outcomes.

A majority of legislative discussions about the value of NSA programs have been highly secretive. The Senate Intelligence Committee lists 56 hearings in 2013, with only three open to the public. The House Intelligence Committee has had fewer than ten open hearings in 2015.

Such lack of transparency has undermined public confidence. This has steadily continued through 2014 and 2015.

Australia turns a blind eye

It appears too many Australian policymakers have been sleepwalking – or blissfully ignorant – when faced with the historical backdrop of such hard-edged, internationally focused evidence.

Without doubt, terrorism is a serious problem. It is clear that the Australian government must undertake precautions to protect citizens. But mandatory metadata retention remains a radical enterprise which has been poorly justified, has numerous limitations and does not offer clear-cut or immediate counter-terrorism answers.