At least 80 government authorities, from federal and state law enforcers to departments and local councils, are using legal loopholes to lodge 350,000 requests a year for access to Australians’ telecommunications metadata. These requests are made mostly without warrants and often without external oversight, sidestepping the strict national access regime established controversially three years ago.

The practice was revealed during a parliamentary inquiry that Home Affairs Minister Peter Dutton is demanding be cut short, to allow him to accelerate new powers for security agencies to access more data, including information currently protected by encryption.

Dutton spoke privately to the chairman of the parliamentary joint committee on intelligence and security (PJCIS), Liberal MP Andrew Hastie, earlier this week and has now written to him, urging him to get the committee to move quickly.

Prime Minister Scott Morrison weighed in on Thursday, pressuring the PJCIS to stop deliberating and endorse the legislation.

“This is a bill that is before the parliament that I want to see passed in the next fortnight,” Morrison said. “I would urge the committee to complete their review as quickly as possible. Our police, our agencies need these powers now and I would like to see them passed. In fact, I would insist on seeing them passed before the end of the next sitting fortnight.”

The inquiry by the PJCIS has been told dozens of agencies are routinely bypassing the existing restrictions in the Telecommunications (Interception and Access) Act, which permits only 22 security agencies to request access to basic unencrypted data.

Instead, many more across government are requesting subscriber details, including names and addresses and other details such as phone records, call durations and locations, using alternative statutory provisions that allow them to bypass the act’s restriction – including, potentially, agencies among the 22 listed. These provisions allow them to access the information faster and without the oversight of traditional watchdog organisations.

The Communications Alliance, representing internet service providers, other private communications companies and some government agencies, told the PJCIS last week that the requests were putting service providers under pressure because it was increasingly difficult to determine whether they were lawful.

The Communications Alliance fears the new proposed legislation covering encrypted communications will make the situation worse.

It would give new and more extensive access to personal information, including the encrypted information on smartphones and other electronic devices. Access notices would be issued and warrants served in secret.

The proposed legislation is designed to help security agencies prevent terrorism and combat serious crime, which is increasingly being organised via encrypted communications.

These security agencies say that, without it, they can’t keep up with changing technologies and that Australia’s national security is at risk.

It would give them new and more extensive access to personal information, including the encrypted information on smartphones and other electronic devices. Access notices would be issued and warrants served in secret.

But a wide range of other organisations warn that without very careful consideration, the legislation could create more problems than it solves.

Communications Alliance chief John Stanton said a member survey had found at least 80 government bodies had requested unencrypted data by alternative routes using a different law, the Telecommunications Act.

“That’s an everyday occurrence, roughly in the order of 350,000 times a year,” Stanton said.

He said the list provided “might not be complete”.

The bodies included Centrelink; the former department of immigration and border protection, which is now part of Home Affairs and has carriage of the encryption bill; the Australian Border Force, and other law-enforcement and anti-corruption bodies; the Australian Securities and Investments Commission; the Australian Tax Office; Australia Post; and state-based integrity bodies in the racing and taxi industries.

The list also included the federal departments of defence, agriculture, families, housing and community services, as well as state departments of health, employment, fair trading, fisheries, workplace safety, transport and other regulatory authorities.

The Brisbane City Council and the Fairfield and Bankstown councils in Sydney’s west and Rockdale council in Sydney’s south were also listed.

Stanton said: “We have seen some authority creep, I guess you might call it, in the period since the data retention regime came into place.”

Recalled last week to expand on his remarks, he said the Communications Alliance had raised its concerns with both the Communications Department and the Attorney-General’s Department more than two years ago.

“They simply said: ‘Well, if it’s provided for under law then there’s nothing we can do about it.’ ”

Under section 313 of the Telecommunications Act, service providers are required to give agencies all “reasonably necessary” help in enforcing Australian criminal law, assisting in enforcing foreign law, protecting “public revenue” and safeguarding national security. Access in relation to civil proceedings is not allowed.

Section 287 overrides privacy protections in life-threatening situations.

But section 280(i)(b) provides the biggest loophole, allowing for access “if in any other case the disclosure or use is required or authorised by or under law”.

The act was passed in 1997 under the Howard government.

Similar generic wording is being included in new legislation to establish the new Office of National Intelligence. It would allow some agencies to bypass the Privacy Act in exchanging people’s personal information.

For serious offences, surveillance and interception warrants must be authorised under the Telecommunications (Interception and Access) Act. That authorisation can be given by a judge or a member of the Administrative Appeals Tribunal.

At a hearing of the PJCIS inquiry last week, shadow attorney-general Mark Dreyfus asked the Commonwealth ombudsman, Michael Manthorpe, if he thought using these alternative pathways was legal, given parliament had provided a specific regime for access.

“I don’t know whether we’d want to cast doubt on whether it’s legal or not here,” Manthorpe responded, offering to look into it. “I can see there’s a question about whether it’s appropriate or whether it’s what the parliament intended, but I wouldn’t want to go so far as to cast a view about its legality or otherwise this morning.”

Manthorpe confirmed those metadata requests fell outside his powers of review, which were restricted to those made under the Telecommunications (Interception and Access) Act.

Communications Alliance security expert Patrick Fair said the notices asking providers to hand over metadata under the new legislation could be issued with or without a warrant, as now.

In relation to encrypted metadata, it would be difficult for providers – and notice issuers – to determine their obligations.

“When trying to judge proportionality or reasonableness, you have no reference point other than the very wide criteria that was inserted into the legislation after consultation,” Fair said.

“It pretty much gives the issuing officer a blank cheque to form a view that in some context at some time there may be a relevance and a benefit to law enforcement for the notice to be issued.”

Security and law-enforcement agencies have been seeking to reassure Australians that the bill will not give them carte blanche to access mass personal data.

Australian Federal Police Commissioner Andrew Colvin told the committee police would not have “unfettered and random” access. It would be carefully targeted and any access to content – as opposed to metadata – would still require a warrant.

Addressing the committee last month, the director-general of the Australian Security Intelligence Organisation, Duncan Lewis, said the bill was about “engagement and co-operation”.

But the Communications Alliance complained that while some of its member organisations had been consulted during drafting, the peak body representing all of them had not.

Two key national security watchdogs, the Commonwealth ombudsman and the Inspector-General of Intelligence and Security, both gave evidence that although they had input at earlier stages they were not formally notified when an exposure draft of the bill – in which provisions had changed – was completed.

The IGIS, Margaret Stone, said her office – which scrutinises the work of 10 intelligence agencies – found out about the draft through the media.

Ombudsman Michael Manthorpe, who has oversight of other law enforcement agencies, described a similar process.

In contrast, the Home Affairs Department said Australia’s “Five Eyes” defence partners – the United States, Britain, Canada and New Zealand – were “kept informed” and received the exposure draft.

Both the IGIS and the ombudsman said the proposed legislation to let agencies legally crack encryption lacks specific detail of oversight provisions – it fails to spell out their monitoring roles – and needs extensive revision and improvement.

Stone said the legislation seemed to assume her organisation would be a “universal fixer” but without specific mention there could be legal doubt over whether she could scrutinise intelligence agencies’ activities regarding encryption at all.

In evidence to the committee a month earlier, Home Affairs departmental secretary Mike Pezzullo had downplayed committee members’ concerns about using internally approved “notices” to make requests, rather than externally approved “warrants”.

“If we were to say to you ‘that notice is a warrant’ and through an incantation and the sprinkling of some magic dust on it, all of a sudden greater oversight was achieved – it’s the same person: it’s the attorney-general of the Commonwealth rigorously discharging … their ministerial responsibilities,” Pezzullo said.

A month on, Stone did not refer specifically to Pezzullo but her differing view on the importance of precision regarding authorisation and oversight was clear.

“This is not abracadabra,” Stone said. “It’s not an incantation. It has to be real.”

Further, she said other missing provisions in the legislation could leave her unable to do the job properly, even if authorised.

For example, the bill did not require security agencies to notify her when they had issued a notice seeking access to encrypted information – access that might involve breaking another law – meaning her office would be in the dark when trying to review activities. She queried the secrecy provisions.

“I’m very concerned that we’re not looking for a needle in a haystack without being given something in the way of a magnet to draw it out,” she said.

ASIO chief Duncan Lewis had used exactly the same analogy a month earlier while arguing for the powers, saying trying to find “an infinitesimally small amount” of data without providers’ help was “similar to using a pair of precision tweezers to extract a needle from a communication haystack”.

Contrary to those suggesting security agencies wanted “backdoor” access to people’s encrypted devices, Lewis said they wanted to “go in through the front door”.

“The haystack, just to continue that analogy, is of no interest to us,” he said.

But dozens of other organisations have warned that without careful consideration of the proposed changes, the whole system – Lewis’s “haystack” – could become more vulnerable to cyber attack.

Last Friday, Human Rights Commissioner Edward Santow said: “A communications provider can be required to assist a government agency secretly by doing an almost limitless range of acts or things, such as to safeguard the interests of Australia’s national economic wellbeing. That goes much further than combating crime and it increases the risk of a disproportionate human rights impact.”

Last month, Duncan Lewis said a “small number of people” were using modern-day technologies to conceal activities that threaten Australians’ safety and security. Currently, it was impossible for security agencies to intercept and read their messages.

Lewis said all communications in Australia were expected to be encrypted within two years, so access laws had to change.

This week, Peter Dutton used the arrest of three men suspected of planning a terrorist attack in Melbourne to demand the PJCIS inquiry be abandoned and the legislation rushed through parliament to give security agencies urgent access to encrypted information.

Dutton and other ministers also accused the Labor Opposition of obstructing the process and breaking with the usual bipartisan approach to national security, something Mark Dreyfus rejected.

On Thursday night, the PJCIS pushed back, issuing a statement saying it would hold another hearing next week to hear arguments about urgency.

“Since 2014, the committee has considered 15 substantive national security bills and made over 300 recommendations for amendment, all of which have been accepted by government,” chairman Andrew Hastie and his Labor deputy, Anthony Byrne, said.

“These reports have been carefully developed to ensure that new powers are proportionate and appropriately balanced with human rights and privacy, and that commensurate oversight and accountability is provided.”

The PJCIS has special status and is entrenched in legislation because of its important watchdog role in scrutinising national security laws.

It must include six members of the house of representatives and five senators, with a majority from the government. By convention, all of them come from the Liberal and Labor parties.

As part of her crossbench deal to form government, then prime minister Julia Gillard allowed the former intelligence analyst turned whistleblower and now independent MP Andrew Wilkie to join, but his tenure ended with hers.

In his October evidence, Duncan Lewis endorsed the committee’s role in the “robust accountability framework” surrounding security agencies.

“Together with the [IGIS] and other bodies, this committee provides assurance to the Australian community with regard to the work that we undertake to ensure our actions are fair, proportionate and lawful,” Lewis said.

The Communications Alliance’s John Stanton believes it has more work to do.

“We certainly hope that the PJCIS will be allowed the time it needs to very thoroughly look at the provisions of this bill,” Stanton told the committee last month.

Not if Peter Dutton and the prime minister prevail. They say time’s up.