It was the 1960s and Richard Norton-Taylor had just finished Oxford with a third-class degree in history. “What would you like to do?” his tutor asked him. Norton‑Taylor shrugged. A few days later, he found himself arriving at a grand building in Carlton Gardens.

It was the headquarters of MI6. Upstairs a figure with a military bearing in a pin-stripe suit greeted Norton-Taylor. The person said: “Don’t tell anyone you have come here.” Another interview followed. It featured questions on the policy of Harold Wilson’s Labour government towards Southern Rhodesia.

Norton-Taylor decided that a career in spying wasn’t for him. What the world of espionage lost, journalism gained. The two professions are not so different: both involve gathering secrets, with the reporter’s job to “reveal them”, Norton-Taylor writes, in his penetrating memoir, The State of Secrecy.

The “fetish” for keeping the public ignorant is absurd and counterproductive, he argues

Over the next 50 years, Norton-Taylor reported for the Guardian on Whitehall and the intelligence and security agencies. I was a colleague. He argues that a culture of excessive secrecy down the ages is responsible for many of Britain’s current ills. There is, he says, a peculiar British willingness to put up with official secrecy, not seen in the US or in Europe.

The “fetish” for keeping the public in ignorance is absurd and counterproductive, he argues. It led to disastrous wars (Afghanistan, Iraq, Libya) and to egregious acts by the spooks (MI6’s collusion with the US in the abduction and torture of terror suspects, denied for years). Secrecy doesn’t enhance national security. It radically undermines it, he thinks.

Facebook Twitter Pinterest Richard Norton-Taylor in 1972. Photograph: Ken Saunders/The Guardian

Norton-Taylor is especially scathing about the “sclerotic” and “secretive” ministry of defence. It has wasted billions on “ill-conceived” weapons irrelevant to modern conflict. He’s critical, too, of the foreign office, which won’t release files on colonial-era abuses. They include the torture of Mau Mau insurgents in Kenya and the murder of Malayan villagers in 1948.

Why do we accept this? The answer, the memoir suggests, is a naive belief that the UK state is a force for good. That, and a love of arcane ritual. Peers and MPs make up parliament’s intelligence and security committee. It oversees the secret agencies. A precondition of sitting on it is to swear a privy council oath. Members promise “to keep secret all matters… revealed unto you”.

It is this mystical culture that possibly explains why the committee didn’t leak its own recent report into Russian meddling ahead of last year’s election. Boris Johnson cynically sat on it. You would have thought at least one MP would put the public interest above medieval vows. And yet none did. Voters are still in the dark.

There is a tradition of journalists willing to suck up to British intelligence and publish spook-friendly stories – sometimes wildly inaccurate ones. Norton-Taylor was too independent and clear-thinking for that. He maintained good contacts in this crepuscular world, though. Often MI5 and MI6 officers told him to keep digging. His long relationship with the spies was “cat and mouse”, he writes.

Along the way there was fun to be had. Norton-Taylor’s first Guardian job was in Brussels in the 1970s, a congenial and gossipy city where everyone spied on everyone else. In 1986, he spent six weeks in Australia covering the trial of Peter Wright, the former MI5 officer whose memoir, Spycatcher, Margaret Thatcher tried to ban.

The cabinet secretary, Sir Robert Armstrong, defended himself against charges of lying by saying he had merely been “economical with the truth”. For Norton-Taylor and other reporters sitting on the Sydney press bench, there were lunches of freshly caught fish washed down with good chardonnay. Best of all, Thatcher and the establishment lost.

As the Soviet Union wobbled, Norton-Taylor flew to Moscow in 1990 to interview George Blake, the MI6 spy turned KGB agent. He travelled with Pat Pottle, a peace campaigner who had helped Blake escape from Wormwood Scrubs prison. Blake’s flat was filled with works by Lenin, 55 volumes, as well as books by Trollope and a life of Gladstone.

Facebook Twitter Pinterest George Blake, the MI6 spy turned KGB agent, whom Norton-Taylor interviewed in 1990. Photograph: Alexander Natruskin/Reuters

Blake said he had got on well with the Cambridge spy Donald Maclean. He was less keen on the “alcohol-fuelled” Kim Philby, Norton-Taylor recalls. Double agents – traitors and defectors – could be useful to both sides, he says. Oleg Gordievsky, the most celebrated KGB defector, told British intelligence the Politburo was convinced the US and UK were about to launch a preemptive nuclear strike.

The State of Secrecy is good on Blair-era misconduct. It takes aim at Sir Richard Dearlove, the MI6 director at the time of the Iraq invasion who gave Tony Blair wrong intelligence about weapons of mass destruction. Norton-Taylor isn’t universally critical of spy chiefs. He praises Eliza Manningham-Buller, the former head of MI5, whose openness and honesty he grew to respect.

The State of Secrecy is an entertaining and timely book, written by a fine reporter who has made a habit of speaking unwelcome truth to power.

• The State of Secrecy: Spies and the Media in Britain by Richard Norton-Taylor is published by Bloomsbury (£25). To order a copy go to guardianbookshop.com. Free UK p&p over £15