In this story

On Tuesday of the last sitting week of parliament, formal notification came that the government’s long-delayed data retention legislation would be up the next day.

Finally, after three months’ consideration by the powerful Parliamentary Joint Committee on Intelligence and Security (PJCIS), and more than three-dozen recommended changes, debate would begin on the bill that some consider a major threat to the privacy and civil liberties of all Australians.

But about 20 minutes later, notification came that it would not be up.

And each day, the pattern repeated. “It was put back on and then taken off,” says Greens senator Scott Ludlam, “Something odd was going on.”

Shorten and Labor’s Past experience has shown what a stumbling block principle can be in matters of national security.

It appears now that the government never really intended to go ahead with it. The legislation was not even drafted. It was put on the notice paper each day because Prime Minister Tony Abbott had said he would have the bill in the parliament that week.

Which only goes to show Abbott’s desperation to look strong on something that might distract from the government’s domestic policy shambles.

Elsewhere in Parliament House, the leaders of the Labor Party had quite the opposite concern. Some of the troops were threatening to provide a distraction from the government’s woes by straying from the expedient bipartisanship that has been a mark of the national security debate over the past 18 months.

The deal the major parties had done to pass the Telecommunications (Interception and Access) Amendment (Data Retention) Bill 2014 was looking decidedly shaky.

There were a number of issues. Some in Labor saw no need for the legislation at all. Like the Greens, like the civil liberties organisations, they worried about the capacity for abuse inherent in private telecommunications companies being required to store data at the behest of government.

Others had more specific concerns. The government had agreed to all but one of the changes driven through the PJCIS by shadow attorney-general Mark Dreyfus and shadow communications minister Jason Clare. But could the government be trusted to adequately incorporate those changes into the legislation?

Labor had been dudded before, notably on provisions of a previous tranche of the security package, the Foreign Fighters Bill, where they only realised after the event that the legislation had not matched what they thought they had agreed to.

They would not believe the government this time until they’d had a chance to examine the actual draft legislation. At time of press, they had yet to see it.

1 . Protecting journalists

There was yet another area of concern, relating to recommendation 26, the one on which government and Labor members of the security and intelligence committee had been unable to agree.

The issue was the protection of journalists and their sources.

The Labor members of the committee were insistent that journalists’ metadata should not be accessed except with a warrant. The Libs wanted a lesser level of protection. An uneasy compromise was reached by the committee.

“The committee acknowledges the importance of recognising the principle of press freedom and the protection of journalists’ sources. The committee considers this matter requires further consideration before a final recommendation can be made,” the report said.

“The committee therefore recommends that the question of how to deal with the authorisation of a disclosure or use of telecommunications data for the purpose of determining the identity of a journalist’s source be the subject of a separate review by this committee.”

It would report back within three months. But by then, the legislation would be in place.

There are serious questions being asked inside Labor, if not by Bill Shorten, then of him: what if the government refused to make the relevant change after the bill passed?

Worse still, what if the Greens or some other minor player put up an amendment calling for the same journalists’ protection that the Labor members had fought so determinedly for in the committee?

Would Labor vote it down, having missed the opportunity to strike the blow themselves? And if they voted it down, how could they go back to the committee and insist on it?

It was messy. Three meetings of the shadow cabinet discussed the issue. Then the full caucus did. It was decided they would defer the matter until they had seen the legislation.

2 . Stumbling block

But Labor remains nonetheless facing the prospect of being wedged by its own regard for freedom of the press.

The bill is expected to be brought on this week. Watch for the Greens’ only representative in the lower house, Adam Bandt, to move the embarrassing amendment.

This is just the sort of scenario the Shorten Labor Party has tried hard to avoid. Past experience has shown what a stumbling block principle can be in matters of national security.

They will recall the way John Howard wedged them. Long memories will go back to the all-night session of December 2002, when they opposed new draconian powers Howard wanted to give the Australian Security Intelligence Organisation to hold and question terrorist suspects.

They will recall him leaning across the dispatch box and accusing them of “security vandalism”, and baiting a flush-faced Simon Crean: “The redder his face got and the more personal he became, the more I knew what I had said was right. He is weak on the issue. He does not have the courage to tell his party what is in Australia’s interests.”

The electorate, in those war-fevered days, sided with Howard, notwithstanding his hypocrisy in pulling the entire bill, thus denying the Australian people the protection he claimed to offer.

Labor does not want to fight on the battlefield of national security, although the voters now are calmer about it. Even when the government is out of step with public opinion, Shorten Labor remains in lockstep with the government.

To cite one recent example: when the prime minister announced the commitment of another 300 troops to Iraq a couple of weeks ago, Shorten happily concurred, saying: “the commitment most certainly likely accords with Labor’s principles and the scope of activities that we’ve outlined.”

3 . Low public support

The public isn’t going along with it, though. The most recent Essential Media survey of voter opinions found just 36 per cent approved. Fifty per cent were against it. Just 12 per cent of people thought it would make Australians more safe. Thirty per cent thought it would make us less safe. Forty-seven per cent thought it would make no difference.

But Labor wants the punters to see nothing between them and the government in this area.

It drives civil libertarians to despair.

Stephen Blanks, president of the New South Wales Council for Civil Liberties, is appalled not only by various provisions in the multiple tranches of legislation the government has introduced over the past eight months, but at the fact that the Labor opposition had “rolled over” to pass them.

He reels off some: the extraordinary powers given to ASIO to access individuals’ computers, and through them networks of computers, powers “so broad and unprotected that ASIO could plant spyware, for example”.

The exemption of ASIO officers from prosecution under broad areas of criminal law, and the criminalising of reporting on their special operations.

The fact passports can be cancelled without notice or right of appeal and “that customs officers or any state official can stop people at the airport and not charge them for an offence and not be accountable in any way”.

He has many more issues, but you get the drift.

“I’m afraid I’ve got to the point where I don’t think Labor believes in anything,” says Blanks. “Everything is tradeable. It is a political tactic to enable them to say there is not a cigarette paper of difference between them and the government on this stuff.”

4 . Terrorism concerns

That’s a bit tough. The fact is we do have a real problem here. More Australians have gone off to fight in Iraq than went to the Spanish Civil War, or to Afghanistan. There is legitimate concern about returning terrorists, and home-schooled terrorists.

To a significant extent, Labor does share the concerns of the government. And it has pushed for changes where it has deemed the government has gone too far.

Granted, it has not gone too hard. Many Labor members – and no doubt some Liberals – have metaphorically held their noses while agreeing to some things. But Labor has pushed for changes. It has just done so very, very quietly.

Data retention laws are a strong case in point. The shell of a bill that the government sent off to the PJCIS was in many ways an improvement on the existing regime of data retention and access, and the recommendations that came out of the committee made it better still.

The fact is, our metadata has long been harvested. In 2012-13, according to the PJCIS report, more than 80 commonwealth, state and territory agencies made requests for metadata, resulting in some 550,000 “disclosures”. That number is growing like Topsy.

The bulk of requests for access to metadata held by telecommunications companies, however, came from state and federal police. The Australian Competition and Consumer Commission and Australian Securities and Investments Commission also are heavy users. ASIO is, too, although the spy agency’s numbers are not public.

The magnitude of those figures invites a number of observations. The first is that this clearly is not substantially about counterterrorism operations. As Tony Abbott said in his “six flags” national security speech a few weeks ago, ASIO has about 400 current counterterrorism investigations on foot.

Yet in the same speech he said: “The government’s Data Retention Bill – currently being reviewed by the parliament – is the vital next step in giving our agencies the tools they need to keep Australia safe.”

That’s not really true. They already have those tools, as the figures show, and as Communications Minister Malcolm Turnbull said when introducing the data retention legislation on October 30 last year: “It does not expand the range of telecommunications metadata which is currently being accessed by law enforcement agencies. It simply ensures that metadata is retained for a period of two years.”

Abbott’s suggestion that there would be an “explosion in unsolved crime” unless the new metadata laws were passed also was false.

That is not to suggest that metadata is not a powerful tool against serious crime. The question is which agencies should be able to use it against which people under what circumstances. As of now it is pretty much open slather. All sorts of minor enforcement agencies, from local councils to the RSPCA, can access metadata without real oversight.

The original bill sought to address some of this. While it did not alter arrangements for access, it did restrict the number of agencies to have it by about three-quarters. Oddly, two of the agencies cut out were the two that dealt with corporate crime, ASIC and the ACCC. The PJCIS reinstated them.

5 . Definition of data

The definition of what data can be accessed was also tightened. Agencies will not be able to access people’s browser histories. They will not be able to track mobile phones while they are not in use.

The committee flagged a number of new safeguards against misuse, including oversight by the ombudsman and the PJCIS itself, which to date has not been able to look into agencies’ operational matters.

Indeed, even the Greens concede the intelligence and security committee’s 38 recommendations for change did a lot to improve the legislation.

Says Ludlam: “We don’t think the bill should be presented at all, but there are some important proposals out of the committee that should proceed even if the data retention bill were pulled.”

But that is unlikely to stop him and his party from seeking to embarrass Shorten and Labor on the journalists’ provision. It may not even stop some of the ALP’s own members from causing embarrassment.