More than 700 government files, some classified top secret, were released to the public this week at the National Archives in Kew. Selecting what is opened up in this biannual event (documents are held secret under what is known as the “30-year rule”, although that is gradually being reduced to 20 years), however, seems to be entirely random. Apart, that is, from obvious efforts by the Cabinet Office to get some easy headlines.

Files highlighted in this latest batch include European plans in the early 1990s for economic and monetary union and John Major’s struggles to get the Maastricht treaty, which established the European Union, through the Commons. The archives’ press release helpfully refers to unsuccessful attempts by Jacob Rees-Mogg’s father, a former editor of the Times later appointed to the House of Lords, to challenge the treaty in the courts.

The files include juicy titbits, one appealing to the never-ending fascination with the Cambridge spy ring – confirming, for example, that the then home secretary, Henry Brooke, did not tell his prime minister, Sir Alec Douglas-Home, that Sir Anthony Blunt, surveyor of the Queen’s pictures, had confessed in 1964 to spying for the Russians. There are also files from the Foreign Office’s secret “Information Research Department” referring to MI5’s concerns about the BBC’s use of “extremists”.

Most of the files released on Tuesday have passages or whole pages redacted by assiduous Whitehall weeders. The full list of files carefully scrutinised shows that many entire files have been retained, some temporarily, most indefinitely.

They include files from the 1980s and early 1990s on Cyprus, on biological and chemical weapons policy, and on “developments in the European Community”.

Files that have been withheld cover topics such as John Major’s visit to Oman and Saudi Arabia in 1993; plans to close Gibraltar’s dockyard; immigration rules; “arms sales and military assistance to Middle East countries”; and subversion in the UK in 1968. They include a 1963 report on “Unavowable Information Services of Her Majesty’s Government Overseas”, and a 1966 report on “Enquiries about the background of people coming into contact with ministers”.

Curiously, a report on a visit by the US president Richard Nixon to Britain in 1970 has been retained for 56 years. A file dated 1992-93 on the Lockerbie bombing, the Pan Am disaster of 1988, has also been withheld.

Even more curiously, four files relating to the Scott arms-to-Iraq inquiry have been retained. The judge-led public inquiry that was bitterly opposed by ministers who had been involved in supplying Saddam Hussein with equipment he used for his weapons of mass destruction programme, heard devastating evidence. It included the Thatcher government’s secret decision to supply Saddam with equipment that could be used to manufacture weapons after he shelled the Kurdish town of Halabja in March 1988 with gas bombs, killing an estimated 5,000 civilians and maiming thousands more.

Geoffrey Howe, the then foreign secretary, drew up a paper pointing to “major opportunities for British industry”. But he was terrified his plan to increase British arms exports to Iraq, secretly agreed by the government, would be leaked. “It could look very cynical if so soon after expressing outrage about the treatment of the Kurds, we adopt a more flexible approach to arms sales,” noted one of Howe’s officials. The government duly adopted a more liberal approach to exports to Iraq but kept MPs and the public in the dark – a policy, Richard Scott remarked, that was even more cynical.

The Cabinet Office, which is responsible for the release of the records at the National Archives, declined to comment on the record when I asked, specifically, why files on the Scott inquiry and Lockerbie were being withheld. Whitehall’s traditional response when asked about archives is to point to the statute that allows government departments to keep back files relating to security, defence, international relations and personal data relating to living individuals.

It is not quite like that. For there is one section of the (inappropriately named) Public Records Act 1958 that gives Whitehall carte blanche in deciding what files to release and what to retain. Under section 3 (4) of the act, Whitehall departments can hold on to documents indefinitely for “administrative purposes” or “for any other special reason”. The department concerned informs the minister responsible for the archives, the culture, media and sports secretary, now Jeremy Wright. He appoints an advisory council (whose members include a former top Ministry of Defence official and former British ambassador). That body has not had the inclination, and the staff of the National Archives have not had the clout, to argue with Whitehall departments, and in particular the Cabinet Office, the powerful bastion of official secrecy at the heart of government. Until and unless they do, or the act is reformed, we will continue to be prevented from knowing about, and learning from, our recent history.

• Richard Norton-Taylor writes for the Guardian on defence and security