The British government has an unusually strong penchant for secrecy, part of a culture of superiority among those born to rule. It becomes especially clear in the release of government papers via the ‘thirty-year rule’, so often countered by the needs of national security. This week we see another example: the Daily Maverick reports that British dealings with Saudi Arabia back in the 1990s have been withheld from public scrutiny. The strong suspicion is that this links to more recent and controversial arms sales relating to the war in Yemen.

Even stronger state control may well follow the recent general election, making it much more important that high-quality investigative journalism survives and thrives. openDemocracy is already a major player in this regard, perhaps one of the reasons that The Sun included it in its short-lived pre-election ‘revelation’ of “an extraordinary network of hard-left extremists pieced together by former British intelligence officers” that had Jeremy Corbyn at its centre.

More seriously, the election result has given us a Conservative Party in Westminster with two distinctive features: it has moved decidedly to the right, even further than it did in the Thatcher era; and it has an eighty-seat overall majority, enabling further moves in a neoliberal direction once the annoyances of Brexit are out of the way.

The Tories under Boris Johnson might relish that kind of future, but the more thoughtful among them could worry that the neoliberal era, that forty-year turbo-capitalist experiment, may well be comfortably past its prime even as society’s elites do their very best to ensure its survival against the odds. In these circumstances we should expect that dissent against the government will grow and the need to maintain control will then express itself in renewed secrecy of actions and firmness of unaccountable governance.

This is why investigative journalism is needed even more than before, and two first-class additions to the limited library of competent books in this field will be published later this month. One is Phil Miller’s eye-opening account of the growth of one of the most secretive of the UK’s privatised military companies, ‘Keenie Meenie: The British mercenaries who got away with war crimes’ (Pluto Press). It concerns Keenie Meenie Services Ltd, which operated from 1975 to the late 1980s, before being subsumed into Saladin Security. KMS described itself as “the original company to offer specialist security services in difficult and high-risk areas of the world”.

Keenie Meenie warrants an article on its own later in the month, but of more immediate relevance in the wake of the election result is ‘The State of Secrecy: Spies and the Media in Britain’ (I.B. Tauris). This is a hugely welcome contribution from Richard Norton-Taylor, the long-serving security specialist on The Guardian, providing a richly informed discussion of the relationship between the British security system and the press.

Three initial points to make about ‘The State of Secrecy’ are that it is part memoir, part analysis; it reflects Norton-Taylor’s early career experience covering Brussels in the early 1970s; and is seriously readable yet has sufficient footnotes to encourage the reader to look further.

Benefitting from his experience of many whistle-blowing episodes and combined with his own work of bringing out into the open all too many examples of government incompetence and cover-up, Norton-Taylor draws a highly informed picture of governments’ single-minded determination to maintain narrative control. His chapters on Whitehall tactics and the techniques of the security agencies are as good an account as you will find anywhere.

Ranging over a remarkable spread of examples he highlights consistent incompetence, especially in MI6, which an enduring culture of superiority has made far easier to hide. Beyond the security agencies – MI5, MI6, GCHQ and the rest – he also focuses on the military-industrial-academic-bureaucratic complex and its closed world of the revolving door, well-funded lobbyists, monopoly providers and persistently corrupt export markets.

As Norton-Taylor points out, the litany of failures and cost overruns in military equipment over decades includes the Type 45 destroyers, Chinook helicopters, the Nimrod maritime patrol aircraft, the army’s Bowman communication system, the Watchkeeper reconnaissance drone and another drone that didn’t work when it rained. Perhaps the classic was the army’s new Ajax light tank that turned out to be too big to fit in the Royal Air Force’s new A400 transport aircraft, a programme that itself has been subject to long delays and cost inflation.

One thing that makes life difficult for journalists in the mainstream print media is that the great majority of papers have a consistent right-wing stance in which the UK’s defence prowess harks back to its imperial past. This makes critical coverage of a defence story difficult to get past the editor unless there is evidence of gross incompetence and ‘wasting the taxpayers’ money’ on a grand scale. Proving that is tough in an era of secrecy, especially if a defence correspondent needs the cooperation of the military for information.

As Norton-Taylor highlights, this becomes even more difficult when covering British special forces, where opacity is near-total, with includes a cross-party consensus on maintaining nil parliamentary accountability. Not that this prevents the judicious leaking of pro-SAS stories, usually to the Daily Mail or The Daily Telegraph, helping to ensure their high status in the public eye.

The Labour government of Tony Blair did bring in the Freedom of Information Act in 2000, but it was a decision that Blair later regarded as one of his worst mistakes. Norton-Taylor notes that he wrote in his memoir, ‘A Journey’: