The story of your life in metadata is an open book. It paints a picture of where you went, who you spoke with, how long you were there for. What were you doing talking on the phone to the sexually transmitted infections clinic? What were you doing on the street corner where the man was murdered last night?

Privacy, at its most basic level, is about the right of citizens to be let alone by their governments. At the heart of what is happening right now in Australia is a debate around this very idea. And as the federal government succeeded in passing its data retention laws on Thursday, it is an idea that is being challenged more than it ever has before.

The government’s new law is fairly simple. The bill initially only came to 47 pages of amendments. But it is exceedingly vague and has enormous potential for expansion.

What it does is force certain types of telecommunication companies to store certain types of communications data known as “metadata”. This is information about who you called, or emailed, where you were at the time, what kind of device you were using and how you connected to the internet. It spans all phone and web technology.

This type of telecommunications data has always been accessible without a warrant. But data retention will vastly increase the reach of enforcement agencies into our personal lives, by creating a much larger set of data from which to draw.

The privacy issues at the core of this debate are numerous. The key ones include: whether it is reasonable to collect this data on all citizens for two years, whether it is reasonable to make this data accessible without a warrant, and whether - if it is stored – it will be done securely.

And this data does not exist in a vacuum. It will sit along an already vast reservoir of information that is easily accessible. It is easy for the federal police or other government agencies to send a quick message to Apple and ask for IP or location data for a particular device. It can go directly to Facebook, or to Gmail, or other providers, and ask them to hand over the content. The companies will not always do so, and each company varies, but sometimes they will. You can read some of these messages here.

The metadata also sits alongside an enormous international framework of surveillance across the entire architecture of the internet, revealed in such great detail by Edward Snowden. Australia is a ready and willing participant in this regime, as a partner in the “five eyes” network.

Put together, what this amounts to is a comprehensive set of data that can tell you the story of a person’s life.

The government has consistently put forward the case that we need this measure to stop terrorism and serious crime. A mandatory retention scheme of two years, it argues, is both a necessary and proportionate measure to achieve this legitimate aim.

Despite this, most police agencies could not explain how old metadata they had previously requested was. And the bill did not limit the use of metadata to serious crimes or terrorism, meaning it is still accessible for almost any kind of offence. Although the RSPCA will not be able to access metadata now under the bill, it will be able to ask police to track down all those naughty dog owners who let their pooches poop on the lawn.

The government argued there was no alternative to mass metadata retention, and that this bill was an urgent priority that had to be passed. Neither was true. Enforcement agencies already had a very useful power – known as a data preservation notice – where they could compel a provider to initiate collection on a particular person. Such a measure would allow far more targeted forms of surveillance to occur, when there were issues with crimes. And the urgency of the scheme is also difficult to fathom, given that it has a two-year roll-in period.

Journalists have escaped the very worst of this. The government secured the support of the Labor party after agreeing to an amendment to force agencies to seek warrants when accessing journalists’ metadata.

But while data retention is often seen as an issue just about privacy, it is also tied to freedom of speech. If genuine public-interest whistleblowers fear detection and are deterred from coming forward to journalists, this is what is known as a “chilling effect” on speech.

The bill still fell far short of an exemption. And the warrants will only be contestable by a government-picked “public interest advocate”, rather than the news organisation itself.

And the concession around journalists begs questions: why don’t warrants cover others? Why not lawyers? Why not other professionals? Why not everyone?

The government has found two effective ways of allaying criticism of its scheme. Firstly it used fear of terrorism and serious crime, which has long proven to be an effective motivator. With this simple strategy of decrying terrorism and child sex offenders, it largely neutralised Labor opposition and achieved bipartisanship.

More recently it has put in place a second, and more subtle, line of defence. This week we have seen the communications minister, Malcolm Turnbull, essentially telling Australians that the changes are no big deal, and explaining all the ways they could avoid data retention if they wanted to. This second argument has swayed some journalists, particularly those already satisfied with the amendments relating to their own profession.

The issue of how the data will be stored remains live. The telecommunications providers will be subject to a motley patchwork of federal privacy law. Some of the enforcement agencies that will inevitably access the data have already proven themselves remarkably inept at handling such things.

The federal police accidentally leaked information about their own interception capabilities by failing to redact – ironically – forms used to access metadata. The immigration department (which has sought to be included for access in its new incarnation as the Australian Border Force) was responsible for the largest government data breach in Australian history when it published the names of almost 10,000 people in detention on its website.

And the costs remain a mystery. We do not know what the government will contribute, or how much it will end up costing consumers.

Opposition in parliament has been rare. The Greens senator Scott Ludlam has emerged as one of the few voices in parliament who has consistently taken a stand on issues of liberty and privacy.

He introduced many amendments to the bill during the Senate debates to attempt to ensure greater protections for Australians. The independent senator Nick Xenophon and the Liberal Democratic party senator David Leyonhjelm also sought to change the bill and introduce safeguards and protections.

In the end, the bill passed with the support of Labor. This scheme will happen, and your personal data will be stored.

Once it really gets under way, the question for all Australians is whether they are happy with being an open book.