Illustration: Cathy Wilcox. To combat these threats, intelligence agencies have been handed additional funding and there are new laws on the way that will allow authorities greater power to detain and question jihadists, lower the threshold for police to arrest suspected terrorists and give the federal police greater power to seek control orders. But, for many, it is the proposed "third tranche" of legislation for introduction in Parliament later this year that will raise the bar on telecommunications companies' obligations to retain so-called metadata for two years that is the cause for greatest concern. And those concerns exist in and outside the cabinet. Cabinet heard Communications Minister Malcolm Turnbull complain on Tuesday of waking up to newspaper headlines concerning the government's controversial plan for mandatory data retention.

Turnbull argued to his colleagues the government risked unnecessary difficulties by pushing ahead with the data retention regime without fully understanding the details. After all, in 2012, Turnbull was dead against mandatory retention. "I must record my very grave misgivings about the proposal; it seems to be heading in precisely the wrong direction," Turnbull said when Labor was floating the idea but hadn't given it its support. The cost of data retention for telecommunications providers and the privacy implications for retaining Australians' internet and phone footprint are the two key flash points. In an internal memo cited by Fairfax Media's The Australian Financial Review this week, Optus warned that the government's plan to force it to retain metadata would cost tens of millions of dollars and that it should not have to bear the costs. "Based on work done in 2010 and refreshed in 2012, a data retention regime could cost Optus in the order of $30-plus million to $200-plus million, depending on a range of assumptions about scope and definition," the memo said.

Optus is not alone. Internet provider iiNet estimated that, based on confidential briefings it had with government officials in 2010, it would cost between $60 million and $100 million per year to retain the data they wanted then, with the cost increasing as more data is collected. Turnbull, who does not sit on the national security committee, which has been meeting most days over the MH17 tragedy and where data retention was approved on Tuesday, was not called in to discuss the proposals. The decision to agree "in principle" to data retention was taken by the NSC and leaked before cabinet approval. But, as Communications Minister, Turnbull is acutely aware of the cost and complexity attached to the data retention laws and argued he should have been consulted. After all, Howard government communications minister Helen Coonan was called in to the NSC when legislation to do with her portfolio was discussed, so why wasn't he?

After Tuesday's announcement that a data retention regime would go ahead, the sell by government ministers went from bad to worse. Initially, Abbott suggested, incorrectly, that metadata was "not what you're doing on the internet, it's the sites you're visiting". In subsequent interviews, the Prime Minister, his office and officials clarified that records of individual web pages visited would not be stored but that metadata – including the IP address of a particular web server accessed and the time and duration you spent on it – would be recorded. By Friday morning, this appeared to have changed too. (In many cases, a single IP address might service hundreds of completely different and independent websites. This means that an IP cannot be considered a full web browsing history. When accessing some sites, though, it can be considered a type of history.) What Abbott said was nothing compared with what Brandis then went through in an interview on Sky News that some of the Attorney-General's colleagues compared with Bill Shorten's infamous "I haven't seen what she said but I support what she said" interview. In the interview, Brandis said, more than once, that the "web addresses" people visited would be captured under the proposal but attempted to clarify, confusingly, it would not extend to web surfing.

It can't be both. Reading between the lines, it appears the Attorney-General was referring to IP addresses accessed as opposed to individual web pages. As an exercise in selling public policy, it was a disaster. Coming so soon after the dumping of the repeal of race hate provisions contained in section 18C of the Racial Discrimination Act – a proposal that seemed doomed to fail from the moment Brandis pointed out that Australians had the right to be bigots – it was not a good sign. As one colleague observed, the deputy Senate leader's hopes of replacing Eric Abetz as leader in the Senate have fast receded into the background. And, in the short term, he has another mess to clean up.

By Thursday afternoon and early Friday, Turnbull began to try and clarify the situation created by his colleagues. Australian Security Intelligence Organisation and Australian Federal Police officials also held a news conference in an attempt to set the record straight. Appearing on radio and TV interviews Friday morning, Turnbull said IP addresses accessed and web browsing histories would not be stored as part of any data retention scheme. Who you called, for how long and from what location would be stored. And the IP address assigned to you by you internet provider would also be stored too, he said, so that law enforcement and spy agencies could identify people involved in a crime. Despite this, some, including those in internet industry circles, remained confused. Confidential briefing documents they received in 2010 show that source and destination IP addresses accessed should be included as part of any data retention scheme. Meetings held by Brandis and Turnbull with three Australian telecommunications companies, intelligence agencies and others on Thursday night appear to have changed this.

Australia's federal privacy commissioner, Timothy Pilgrim, also appears to have been left out of the debate before it was approved. "The government's proposal on data retention raises a number of important issues and so it is important that there is an opportunity for public consultation and debate on these proposals once the detail is available," he said on Friday. He said it remained "unclear" exactly what type of information would be retained. "However, there is the potential for the retention of large amounts of data to contain or reveal a great deal of information about people's private lives and that this data could be considered 'personal information' under the Privacy Act," he said.