For decades, hundreds of official documents, including cabinet papers, have been released on 30 December, disclosing decisions that would have been considered unacceptable in a democracy had they been known about at the time. They would be made available at what is now called the National Archives at Kew in south-west London, under the “30-year rule”.

This year will be different. Instead of the usual treasure trove of long-held documents, next week the archives will release only a limited number. Kew says it is changing the way historic records are released by “moving to a more responsive and agile programme of releases”. Its task has not been made easier by the decision to gradually open up records 20, rather than 30 years, old. But the official explanation put out by the archives disguises a fundamental shift in Whitehall that is making a mockery of David Cameron’s early promises of transparency and accountability.

Opening up official information, including historical records, has been downgraded. It is “a very low priority”, say senior figures with intimate knowledge of the issues. While ministers and their civil servants slash the number of staff responsible for weeding through and releasing government documents, they also blame the Freedom of Information Act for costing too much. Yet what is costing – and wasting – money is Whitehall’s increasingly determined attempts to suppress information.

Revealed: Prince Charles has received confidential cabinet papers for decades Read more

Here is an example of the lengths to which the government is going to stop information from seeing the light of day, and the costs involved. Maurice Frankel, director of the Campaign for Freedom of Information, has worked out that Whitehall has so far spent more than £30,000 of taxpayers’ money in legal fees in a four-and-a-half year battle trying to stop the release of the appointments diaries of the former health secretary Andrew Lansley.

Whitehall argues that it would be damaging and embarrassing if it revealed there had been gaps in his diaries. Officials in future would have to fill up their diaries with pointless and unnecessary meetings, it argues. Yet that would be another reason why the diaries should be disclosed, according to the tribunal dealing with the case. They would expose just how civil servants were prepared to waste even more time and money. The government, rejecting the point the tribunal was trying to make, has taken the case to the court of appeal.

We also now know that senior Whitehall officials fought for three years in a failed and costly attempt to prevent British citizens from knowing that Prince Charles has access to cabinet papers. “Obviously it would have been much better if they would have been open on this point,” said the chair of the Commons constitutional committee, Conservative Bernard Jenkin. “But this is the civil service … where they’re still not used to drawing a line between what is secret and what is not secret.”.

The most senior Whitehall mandarin, Sir Jeremy Heywood, the cabinet secretary, claims the Freedom of Information Act has a “chilling effect” on the way his officials operate. The government has set up a commission designed to make it more difficult to obtain official information under the act – which has exposed child abuse, the misuse of MPs’ expenses, unhygienic restaurants and nuclear power station leaks.

Whitehall has also taken responsibility for freedom of information policy away from the Ministry of Justice and placed it firmly in the hands of Heywood’s Cabinet Office, the bastion of official secrecy. And the release of records to the National Archives has been placed under the control of what are called “knowledge management” units set up in Whitehall departments.

Ministers and their officials need not worry. The obstacles in the way are already formidable. The loopholes and exemptions in both the Freedom of Information and Public Records Acts allow Whitehall officials tremendous leeway. The notorious section 3 (4) of the Public Records Act states that documents can be retained by Whitehall departments “for administrative or any other special reason”.

One person’s experience provides eloquent testimony. Nine months ago, Andrew Lownie, an author and literary agent, asked the archives to release a Foreign Office file marked: “Closed extract: Folio 1 from Defectors to and from the Soviet Union, including Kim Philby”. At first, the archives said the information was exempt under section 27 of the Freedom of Information Act, whereby disclosure would be refused if it “prejudices Britain’s relations with another state” or “the interests of the United Kingdom abroad”. By June the archives had changed tack. Now it said the information was exempt under section 40 (2) of the act, covering “personal information”.

In a catch-22, the act allows Whitehall to withhold documents if the intention is to publish them some time in the future. There are about 600,000 Foreign Office files in a “special collection” waiting to be transferred to the archives. It has been estimated it will take 75 years for them to be weeded and sent to Kew. It will also be a long time before there will be a proper, regular release at the archives of major cabinet documents.