The refusal of FOI requests is at its highest level since records began and a Guardian Australia investigation has found systemic problems

The Australian government is refusing access to documents at record rates, aided by a flawed freedom of information regime beset by delays, understaffing and unnecessary obfuscation.

A month-long investigation into the operation of freedom of information (FOI) laws has identified systemic problems causing vast volumes of government information to be kept secret. Guardian Australia has found:

FOI refusals are at their highest level since records began in 2010-11, spiking recently due to the secrecy of the Northern Australian Infrastructure Facility (Naif), the agency that gave conditional approval to lend Adani $1bn in taxpayers’ money. Naif rejected 99.4% of the FOI requests it received.

More than 2,000 FOI requests have taken three months longer than the statutory time frame to finalise, rendering the documents all but irrelevant by the time they are released, if they are released at all.

FOI teams have shrunk in at least 20 government departments or agencies. Meanwhile, the federal government is increasingly refusing to process FOI requests because they are too onerous. The government’s use of “practical refusal” grounds to block FOI requests has skyrocketed to record highs, increasing by 163% last financial year alone.

The regulator, the Office of the Australian Information Commissioner, has been chronically understaffed, despite an expanding remit and a 72% increase in the number of complaints received about FOI. The OAIC was gutted under the Abbott government, leaving it with two-thirds of the 100 staff minimum needed to do its job.

Academics, authors and not-for-profits are being denied the most benign of documents. Journalist and author William Summers has been battling to obtain a copy of Parliament House lunch menus for two months without success because the Department of Parliamentary Services is not beholden to FOI laws. Lockout campaigner and academic Tony Brown was wrongly blocked from receiving a hotelier’s submission to a recent inquiry into Newcastle’s lockout laws because the NSW Department of Industry indicated it had promised the publican it would remain secret, in a submission seen by Guardian Australia.

In the past year, heavy redactions have prevented the public knowing whether Australians are fighting as mercenaries in the bitter conflict in Yemen.

The dealings between Australian companies and sanctioned elements of the North Korean regime have been hidden, as have communications between Australia and the UK about the WikiLeaks founder Julian Assange.

Academics, journalists, crossbenchers and anti-corruption campaigners are now calling for change, urging the government to overhaul Australia’s FOI laws to improve transparency.

Lawyer Peter Timmins, a respected FOI expert and former diplomat, wants to see a comprehensive review of Australia’s FOI laws to make them “fit for the 21st century”.

“The government has been sitting on a review since 2013 when Allan Hawke undertook a review of the act and his first recommendation was there needed to be a full review of FOI legislation, and a rewrite of the act in a way that made it more understandable and accessible,” Timmins said. “That’s never happened.”

Last financial year was particularly poor for government transparency. The rate of FOI refusals was at a record high (17%), and the proportion of requests being granted in full was at its lowest (50%) since the OAIC began publishing data.

The result is partly due to the secrecy of the Naif.

The agency agreed to fully release documents in only one of the 1,340 FOI requests it received in 2017-18, giving it a rejection rate of 99.4%, by far the highest of any agency. Naif was inundated with requests for internal documents in a coordinated Greenpeace campaign over a two-week period last year, and a spokesman said the agency complied with all its legal requirements.

Fees also continue to prove prohibitive for small not-for-profits and resource-starved media outlets. In one recent case, the Australian Conservation Foundation was asked to pay almost $500 for documents showing internal discussions on why climate change was largely absent from the government’s 2015 intergenerational report. It paid the amount, only for 241 of 243 relevant pages to be deemed exempt. The two pages it received – copies of calendar events – were also partially redacted.

Even when documents are released, they are often rendered irrelevant by delays.

Agencies are required to process requests for information within 30 days, but routinely push the time frame out, typically by claiming they need to consult third parties or that a request is complex or large. Data shows delays ballooned significantly in 2016-17, when only 59% of requests were processed within the statutory timeframe, though the situation has since improved.

The Department of Home Affairs, by far the biggest recipient of FOI requests, was particularly prone to delay. It exceeded the statutory time frame by three months or more in 1,990 of the 15,220 requests it received last financial year. The department has now engaged a private “service provider” to help it process FOIs “within a limited scope”, and a spokesperson said it had implemented reforms to help individuals access personal information more easily.

One former OAIC insider, who asked for anonymity because he still works for government, said delays were used deliberately to take the sting out of sensitive documents.

“A lot of the tactics are really just to delay the release, particularly until the heat has gone out of something,” he said.

“When a document is released, even if it does cause a ripple, it’s a ripple that lasts a day. It’s just that in the current age it’s all about managing the message, and FOI is just inconvenient.”

Transparency International believes delays are at least partly caused by staffing losses in departmental FOI teams. An analysis of FOI staffing numbers shows reductions in at least 20 government departments or agencies in the past seven years.

The most severely affected department was the Australian Taxation Office, which has lost 15 FOI staff since 2013-14. The Department of Social Services has lost six FOI staff since 2014-15.

The staffing reductions coincide with the growing use of “practical refusals”, which allow the government block FOI requests if they take up too much of the agency’s resources, or if the applicant fails to properly identify documents. The use of practical grounds to attempt to block FOI requests is now at record levels.

The chief executive of Transparency International Australia, Serena Lillywhite, said timely access to information was important, and increasing refusals were a sign of “inadequate government resourcing to this important element of open government”.

“The fact that they do not prioritise resources for FOI requests reflects the fact they do not regard freedom of information as a priority function for most government agencies,” Lillywhite said.

“A 2017 Australian National Audit Office (ANAO) audit revealed increasing numbers of FOI applications received in contrast to a declining trend in funding for FOI functions and subsequent delay in completions.”

The regulator, the OAIC, plays a critical role in ensuring departments respond quickly and properly to FOI requests. Its presence deters departments from using spurious or flawed reasoning to deny FOI requests, and the regulator can act as a circuit-breaker in disputes.

But the commission has experienced chronic understaffing. The OAIC, when it was established, estimated it would need at least 100 staff to fulfill its role effectively. It has been below that level every year since, dropping to a low of 63 staff under the Abbott government.

The commissioner, Angelene Falk, has repeatedly warned the office’s absorption of a growing workload with a limited workforce was creating “challenges”.

“The workload is increasing, we’ve had increases on freedom of information and privacy year on year, particularly over last three years, and we’ve worked very effectively at identifying efficiencies,” Falk told Senate estimates in October. “But in terms of our ability to absorb, if you like, the downstream effects of new proposals that occur, that is becoming more challenging.”

Australia Institute researcher Tom Swann said he had noticed an increase in delays and rejections, particularly in the past couple of years. Swann, who is responsible for most of the institute’s FOI requests, said most rejections were attributed to too many documents being included in the initial search.

Swann said most agencies were helpful in refining requests, but others seemed to deliberately cast “a very wide net” for documents.

“The result is so many documents they can’t process it,” he said.

Last year, Swann filed a FOI request to the Department of Foreign Affairs and Trade, searching for documents that showed the Australian government attempting to secure the support of foreign governments or foreign investors for the Adani coalmine. After much back and forth, Swann was informed no documents matched his search.

It was later revealed that ministers had in fact written to the Chinese government welcoming foreign investment and promoting the Adani coalmine. So Swann took the case to the information commissioner.

“It’s clear that the information commissioner struggles with the level of resourcing they are given and their internal priorities are not necessarily their own priorities. They struggled with the workload and that’s resulted in that external review is still only just been looked at now.”

Swann said he believes most FOI officers were trying to promote a culture of disclosure but were limited in their ability to do so. Without additional support and given the external reviewer is underresourced, agencies lose their incentive to proactively share information.

“Intentional or not, the incentive for disclosure is being reduced when the chance of getting a slap on the wrist is reduced.”

The crossbench senator Rex Patrick recently introduced amendments to the FOI act, in an attempt to address some of the system’s fundamental failings. The changes failed to win the support of a committee dominated by the two major parties.

Patrick said his amendments were designed to make FOI more user-friendly.

He said FOI had weaknesses as a democratic tool, questioning how ordinary citizens could use the system effectively, if journalists, politicians and researchers struggled.

“One of the problems we have with FOI is it can take a year, maybe more, to get access to anything with any complexity and information has a temporal value to it. What you really want is real-time access.”

His bill seeks to force the government to fill all three roles of information commissioner, privacy commissioner and freedom of information commissioner, which are all currently performed by a single commissioner. Patrick said it would assist applicants seeking FOI reviews in the appeals tribunal and force agencies to reveal their spending on legal advice fighting FOI requests.

“It’s my view that everything that the government does, they do so for public purpose and funded by the public service. And therefore everything that is produced by government belongs to the public.”

Peter Timmins (@foiguru) Senate committee hearing last week on @Senator_Patrick's bill to improve #FOIA reveals two different worlds- some users of the act think things have never been worse; gov witnesses think things are going pretty well.https://t.co/sAIMXSaEJH

The federal government is not the only jurisdiction where basic FOI reforms are failing to get traction.

In NSW, recent amendments to the state’s government information public access scheme enshrined the right of agencies to ask that requests to be made by “snail mail”. The opposition, led by the shadow attorney general, Paul Lynch, had urged the government to compel government agencies to take requests electronically.

• This reporting is supported by the Susan McKinnon Foundation through the Guardian Civic Journalism Trust