The Government has again announced its data retention scheme and again revealed its shortcomings. Once these databases are created your details will just be a subpoena away, writes Chris Berg.

Last week was the second time the Government announced its mandatory data retention policy, and the second time it gave the game away while doing so.

Data retention keeps spinning out of the Government's control.

First, in August, Tony Abbott admitted in a television interview that requiring internet service providers to retain data on their customers' activity was not just about anti-terrorism and national security but could be used to fight "general crime".

This time the mistake was made not by politicians but by the Australian Federal Police commissioner Andrew Colvin.

Asked whether data retention could be used to police copyright infringement, Colvin responded:

Absolutely, I mean any interface, any connection somebody has over the internet, we need to be able to identify the parties to that connection ... So illegal downloads, piracy ... cyber-crimes, cyber-security, all these matters and our ability to investigate them is absolutely pinned to our ability to retrieve and use metadata.

Over the next few days George Brandis, Malcolm Turnbull and Colvin tried to roll this back. Copyright is a civil wrong, not a criminal one, they said. Copyright holders are responsible for bringing legal action against pirates. The AFP isn't interested in civil cases. (This is only partly true. Commercial scale copyright infringement is a criminal offence.)

But here's why Colvin's misstep matters.

Mandatory data retention would create massive new databases of internet users' activity in every internet service provider across the country.

A lot of opponents of data retention have pointed out that this creates a very real risk of unauthorised access. It's hard to keep data secure.

Yet just as concerning is authorised access. Once these databases have been created they will be one subpoena away from access in any and every private lawsuit.

Many people have some residual faith that police and security services are benevolent. After all, their mission is absolutely essential - to protect us. But do Australians have the same faith in movie studios? Their neighbours? Their employers?

After all, it's been undeniable that data retention could help copyright infringement cases ever since the Government included "download volumes" in the list of data it wanted ISPs to retain.

But this is just getting started. Think about how useful mandatory data retention might be in other civil cases.

It would be easy to trace where somebody has been based on the source IP addresses of their mobile phone, as the phone moves from cell tower to cell tower, connecting and reconnecting to the network and internet every time.

In other words, under mandatory data retention ISPs will have to keep records of your movements for two years.

Imagine how this sort of information might be used, for instance, in a workplace relations lawsuit.

Likewise, online defamation cases will be strengthened by records that match IP address to account holder. Do you sometimes comment anonymously on blogs and news websites? Under data retention lawyers could track down who you are months after the fact.

We could go on.

Remember the Government wants this data stored solely for the purpose of future law enforcement investigations. It would be deleted otherwise. It has no business purpose.

Yet not everything about the policy the Government announced last week is terrible.

It was long assumed that data retention would be shoehorned into the existing telecommunications access regime - the regime that allows agencies and authorities from ASIO to the RSPCA to access your phone records without needing a warrant.

Instead, the Government has decided to change that regime.

The proposed bill limits warrantless access to the both the existing set of data, and any future data retained under the new policy, to "criminal law enforcement agencies". Those agencies are the AFP, Customs, state police, and the state anti-corruption commissions. (You can see the list in the explanatory memorandum here, paragraph 197.)

The upshot is that the RSPCA will no longer have warrantless access to phone records. Nor will the Australian Competition and Consumer Commission, the Australian Securities and Investment Commission, or any of the dozens of bodies that have enjoyed such access for years.

They, like movie studios and your neighbours, would have to ask a judge for permission.

I'd guess there was a fair bit of jaw-dropping in bureaucracies across the country when Brandis and Turnbull announced that new rule.

Now, the legislation allows the Government to authorise more agencies at will, so the list could easily expand.

Still it is a striking admission that there has been too much access to too much data by too many bureaucrats for too long.

And that's why the new limits on agency access to telecommunications data doesn't compensate for the threat to civil liberties that is mandatory data retention. Fewer agencies, sure, but with access to a much more complete record of our lives.

One of the clichés of the internet era is that "information wants to be free". But information doesn't want anything, of course. People want information.

Data retention will create vast archives of data about what we have done and where we have been. People will definitely want that.

Chris Berg is Policy Director at the Institute of Public Affairs. Follow him at twitter.com/chrisberg. View his full profile here.