In this story

When she addressed the annual conference of the Royal Australian College of General Practitioners in Perth last week, Health Minister Sussan Ley was already in a hostile environment.

Doctors are angry at cost-saving measures that are putting pressure on their fees. They believe the government has broken promises, used them as a collective cash cow and left them to pass on higher costs to their patients.

Standing at the podium, Ley surprised the GPs by apologising for something else entirely.

Ley revealed that the health department had inadvertently committed a potentially serious breach of the Privacy Act by deliberately publishing supposedly anonymous Medicare and pharmaceutical claims data involving GPs and three million of their patients.

To help health researchers provide better analysis and contribute to health policy, the department had made public “de-identified” records of claims under the Medicare Benefits Schedule and Pharmaceutical Benefits Scheme for a randomly selected sample of 10 per cent of the Australian population.

But it had also included just enough information about its encryption algorithms to enable a competent code-breaker to unravel the jumbled numbers that replaced doctors’ provider numbers and potentially identify them.

Ley did not explain why, when doctors who discover a privacy breach are obliged to alert those affected immediately, the government waited 16 days.

It took analysts at the University of Melbourne’s Department of Computing and Information Systems just a few days to do it.

“Yes, there will always be risks, no matter how slight, around the release of any de-identified data,” Ley told the conference last Thursday morning, as she segued to a nothing-to-see-here confession, five minutes into a half-hour speech. “It’s how we manage these risks when they arise that is important.”

Her department’s risk management is now the subject of considerable discussion across government about how the release of information on the Department of Prime Minister and Cabinet’s data.gov.au website could have been so badly handled.

Ley revealed that the University of Melbourne researchers had notified her department of “a vulnerability” in the encrypted data on September 8 – the researchers say it was actually September 12 – and “that individual healthcare providers could possibly be re-identified”.

Ley assured doctors there were “no provider names in the dataset” and no patient information had been “compromised”.

But the analysts had shown doctor re-identification was possible. They simply chose not to do it.

“It’s certainly something we take seriously and we apologise for any concern this may cause you as providers,” Ley told the doctors.

Ley said they should not be concerned because Attorney-General George Brandis was preparing legislation to make it a criminal offence to publish re-identified records or “counsel, procure, facilitate or encourage” them to be published or communicated.

Revealing the new offence by press release on September 28 – the night before Ley’s speech – Brandis announced it would be legislated when parliament resumes and backdated to that day.

“The publication of major datasets is an important part of 21st century government providing a great benefit to the community,” Brandis said.

But the government also recognised privacy was “of paramount importance”. Strict procedures were applied to ensure data was anonymous.

“However, with advances of technology, methods that were sufficient to de-identify data in the past may become susceptible to re-identification in the future,” he said.

What he did not say was that the law was being changed because those methods had already failed.

And what the health minister did not say the next morning was that with the addition of a few extra obtainable details about individuals, the decrypted health provider data could be used to identify some patients, match up their records, and form a picture of the tests, procedures and drugs that people in the sample had undertaken and been prescribed, revealing their most intimate health information. It would not be easy, but it is possible.

A spokeswoman for the department said that while the academic team had shown the health service provider numbers could be decrypted, “this information has not been published or disseminated”.

“If decrypted, the information would reveal the health service provider number only. Other information would need to be sourced to identify any specific healthcare provider. There is no evidence that this has happened. No patient information has been identified, no patient’s privacy has been compromised in any way.”

The spokeswoman confirmed the data, understood to have been made publicly available in August, had been downloaded 1500 times, 500 times via academic or government domains and the rest by health insurance companies, “consultancies”, “not-for-profit organisations” and other companies or individuals unnamed.

The government can’t say for sure who might already have the data or how they might have used it before the new law takes effect, only that it has not been “disseminated”.

Ley did not explain why, when doctors and others who might discover a privacy breach are legally obliged to alert those affected immediately, the government waited 16 days.

A spokesman for the minister did not return The Saturday Paper’s calls.

The RACGP chair of e-health and practice systems, Nathan Pinskier, says that, at the very least, the time lag indicates a double standard.

“There’s a question around transparent and open disclosure,” Pinskier tells The Saturday Paper. “We’re expected to be held to the highest account but they don’t do the same thing.”

The Melbourne University team who blew the whistle on the health department’s mistake is Dr Chris Culnane, Dr Benjamin Rubenstein and Dr Vanessa Teague. They set out to see how strong the government’s encryption was. It turns out, not very strong.

1 . ‘A lot of detail’

Teague tells The Saturday Paper each anonymised set of data “contained a lot of detail about individuals”.

She emphasises that the patient records themselves could not automatically be decrypted – other specific information would still be needed.

But while those identities were hidden using numbers the team had not been able to decrypt, anyone with their same skills and a bit of extra detail – such as an individual’s age, roughly where they saw a doctor or knowledge of recent procedures or treatments – could piece together the records.

The department says the published data excluded some rare events to help anonymise it.

“It depends on how much is known about the person and how unusual the person is,” Teague says. But she confirmed individuals in the sample were still potentially identifiable.

“The question, if indeed the person is there (in the sample), is how much do you need to know and how unusual do they need to be?”

The decrypted data can potentially be misused, should individuals be identified.

The health history can include tests for diseases, mental health consultations, abortions and signs of chronic illness.

That information could be valuable to an insurance company wanting to increase premiums or refuse cover to individuals based on risk.

An employer might use it to check up on staff. An ex-spouse seeking ammunition for a custody battle might be interested, or a media organisation trading in celebrity gossip, or a stalker, or a blackmailer – hence the new offence.

Based on re-identified provider records, an insurer could examine a doctor’s prescribing history without necessarily having the context of each referral or script.

“That sort of data is really useful as a peer comparison tool between practices,” the RACGP’s Nathan Pinskier says. “… But just to expose it out in the public is of concern.”

2 . ‘Serious questions to answer’

Labor’s shadow health minister Catherine King says the Turnbull government “has serious questions to answer”.

The government’s new law will punish any publication that occurs after last Wednesday but it can’t be sure what might have been privately acted on before then.

“We showed the encryption could be reversed and that was clearly not meant to happen,” Melbourne University’s Teague says.

The team published their findings without revealing the data itself.

“Publishing data can bring great benefits to research but also great risks to privacy,” its report said. “The mathematical details matter: it’s a technically challenging task to understand whether a particular algorithm securely encrypts data or not. Datasets containing sensitive information about individuals clearly deserve more caution than others, and may not always be suitable for open public release.”

Teague says it is “an important thing to investigate”.

Once alerted, the department called in the government’s cybersecurity experts and began an investigation.

It notified the privacy commissioner, Timothy Pilgrim, who has started one of his own.

“The primary purpose of the investigation is to assess whether any personal information has been compromised or is at risk of compromise,” Pilgrim says, “and to assess the adequacy of the Department of Health’s processes for de-identifying information for publication.”

The health department has now asked the university researchers to help improve future encryption processes.

Teague says they will advise on the risks present in what’s already been released – another indication the government can’t quantify those right now – and how data should be treated in future “so we can be confident before we put it up that it doesn’t pose a risk to anybody’s privacy”.

The college of GPs points out that with doctors being shifted to the government’s new digitised e-health system, this was not great reassurance.

“If the government can’t manage one dataset, how can it guarantee it can manage another?” Nathan Pinskier asks.

The transition to the national e-health system has already been slower than the government hoped, prompting it to provide more time and funding for struggling general practices to make the leap.

But the extension until 2020 of the already three-year freeze on the Medicare rebate doctors are paid per consultation has fostered ill will. Advertisements against the measure have been back on television. Doctors are not going without a fight.

New RACGP president Bastian Seidel says that when the government rebate for taking out private health insurance is above inflation, at more than 4 per cent, it’s not logical that the rebate for visiting a GP in the public system doesn’t move with inflation at all.

He says the latest government figures, published last week, show a $177 million annual underspend on bulk-billing – $150 million of which could be used to fund a rebate rise in line with inflation.

But while assuring doctors she wanted to work “in partnership”, Ley also said she needed to save money and their requests were “rarely matched with alternative ways to pay for them”.

She told doctors: “The responsibility for keeping the budget balanced in our relationship can’t be one-sided.”

Australian Medical Association president Michael Gannon had a meeting with Ley last week. He has since written to her, outlining ongoing concerns.

“While the Medicare freeze is not the only issue in the health system, it does represent a speed bump to general engagement,” Gannon says.

GPs are also extremely unhappy about the winding back of bulk-billing incentives for radiology and pathology – now deferred until January – which will likely see a patient co-payment introduced.

And they are livid about a pre-election government deal struck with big pathology companies to force GPs to cap the rents they charge for co-located pathology collection centres.

In the face of government assurances that bulk-billing rates remain high, the college of GPs commissioned its own survey, showing they had fallen from 80 per cent to 69 per cent.

“These are very worrying signals from the general practice community,” Consumers Health Forum chief executive Leanne Wells says.

Seidel says the government’s measures will lead to patients having to pay more without better health outcomes.

“If you are writing policies that are not evidence-based, don’t be surprised by what the consequences are,” he says.

Being effectively ambushed by Ley’s revelation about the data breach didn’t help relations either. Seidel and senior colleagues were informed an hour before the minister’s speech.

The health department says it will keep publishing data to assist health researchers. Private Healthcare Australia’s chief executive, Dr Rachel David, urged that the breach “not derail the process of providing high-value datasets to researchers and stakeholders”. Leanne Wells also says it is crucial to knowing “what works and what doesn’t” but that the incident “points to the need for rigorous assessment of risks to privacy and safeguards”.

The dataset is no longer online. The department says it “will only be restored when concerns about its potential vulnerabilities are resolved”.

Cyber complexities notwithstanding, resolving that may prove easiest of all.