In my recent novel A Delicate Truth, a retired and patently decent British foreign servant accuses his old employers of being party to a Whitehall coverup, and for his pains is promptly threatened with the secret courts. Yet amid all the comment that my novel briefly provoked, this particular episode attracted no attention.

What are secret courts? Why do we need them? To protect Britain's special relationship with the United States, we are officially told; to protect the credibility and integrity of our intelligence services. Never mind that for decades we have handled security-sensitive cases by clearing the court whenever necessary, and allowing our secret servants to withhold their names and testify from behind screens, real or virtual: now, all of a sudden, the credibility and integrity of our intelligence services are at stake, and need urgent and draconian protection.

Never mind the credibility and integrity of parliament and centuries of British justice: our spies come first. And remember, these aren't criminal courts. These are civil courts where anyone attempting to obtain redress for a real or perceived injustice perpetrated against him by British or American secret agencies must have his claims heard and dealt with in secret.

A loyal British soldier sees his comrades being mown down by friendly fire? From now on, he will have to air his grievance in the secret courts. Compensation for the afflicted families? Maybe. But no apology or explanation. That's "policy" or, in plain English, you can't argue.

You have made an enemy of Colonel Gaddafi and are on the run from him? Your wife is pregnant, you constitute no danger to the west, but British intelligence has decided to organise your rendition to Libya as a favour to its old pal the colonel? And you were tortured, and now you'd like redress? Money, yes, you want money, of course you do. Like all your kind, you are grasping.

In reality, Abdel Hakim Belhaj and his wife are offering to settle their case for the princely sum of £1 per person sued plus an apology and a public admission of liability for what was done to them, something they can hold up and share with their friends, some decent gesture of humanity and regret that will provide closure of a sort.

Well, in the view of the British government, Mr and Mrs Belhaj can sing for their terms, because an apology sheds no glory whatever on our special relationship with the United States, or on the credibility and integrity of our intelligence services. MI6 did not render Mr and Mrs Belhaj to Colonel Gaddafi under their proper names, but only as "air cargo". And the plane that flew the hijacked couple to Tripoli was provided by the CIA. And the credibility and integrity of both services are of course paramount, and must be kept that way at any cost.

The true reason for the existence of these gruesome secret courts, I suggest, beyond the desire to protect our state from embarrassment about the nature of our wrongdoing, is twofold: the disproportionate influence of the US/UK intelligence community on our democratic institutions, and the urgent need of our respective political establishments to import a Bush-style secret state to Britain. For Barack Obama, far from dismantling Bush's secret state when he took power, has diligently recrafted and extended it. In consequence, the CIA has become a fully fledged, unaccountable fighting arm, big on extrajudicial killing and derring-do, but short on the hard grind of intelligence gathering, which is where the Brits traditionally believe they have the edge. As part of his deal with the CIA, Obama, on taking office, promised not to rake up the past, which meant not naming or shaming the agency's torturers, or those at the highest level of the administration who had guided their henchmen's work down to the smallest, awful detail. But the past doesn't go away that lightly, and the most pressing task for our secret courts will be to keep the lid on the CIA's unlawful activities under Bush, and our own complicity in them, thereby incidentally clearing a path for them in the future.

These, then, are the two main players in the creation of our British secret courts: our politicians – who seem barely to understand what they have passed into law – and our spies. The lawyers also played a part, but were a sideshow.

We know a lot about the fallibility of our parliamentarians. Of our intelligence services, on the other hand, we know next to nothing, which is how it must be, and how they like it. But whoever they are, and whoever they think they are, it would surely be surprising if their organisations were not prone to the same cockups, coverups and bouts of near-insanity that afflict any other British corporate entity these days, from banking to the press to the National Health Service to the BBC.

Nevertheless, it is the spies, professional charmers and persuaders that they are, and bullies when they need to be, who have so successfully lobbied parliament; it is the spies who will approve and pick and brief the lawyers; the spies again who will produce the witnesses and present the evidence that the luckless claimant may never get to see or question.

Parliamentary oversight? Have you ever spotted those weird recruitment ads that MI6 puts out in our newspapers at our expense? Boiled down, they're all about: how good are you at talking people into betraying their country? I'm not sure how high that particular skill rates on the credibility and legitimacy charts, but one thing that won't have changed in the 50-odd years since I left the secret world, and never will, is the gullibility of the uninitiated when faced with real-life spies. In a flash, all rational standards of human judgment fall away:

Is this man/woman as stupid/brilliant/insightful/witless as he looks, or is it all a charade? Is his moustache real? Is his posh or regional accent real? Does she really wear blue glasses? And then the other bit: does he or she know I'm cheating on my wife? And the spies for their part know their own mystique, and play on it. They can be dedicated men and women, good minds, even very good minds. They can be brave, resourceful, eternally discreet. I never met a better bunch of colleagues. But make no mistake: they know their own legends and foster them, even believe them, and like actors they know the punter is watching them all the time. Their power over you lies in letting you know a little bit, and implying they know a whole lot more; in reminding you of the perils they grapple with, day and night, while you lie in hoggish slumber in your bed. You must take us on trust, they are telling you; or else pay the price when the bomb goes off in the marketplace.

And the trouble is, sometimes, they're right. So the safest thing for your uninitiated politician is to say three bags full, and congratulate himself that he's been admitted to the magic circle, which these days is a very wide one indeed, covering corporations, newspaper moguls, foreign editors, lawyers, doctors and the whole range of candlestick-makers. In the District of Columbia alone, I read somewhere, almost a million non-governmental souls are cleared for top-secret material. One day we'll all either be cleared citizens or unpersons, but until then: be afraid, and go on being afraid till they tell you to stop.

Does anyone remember how we got dragged into the Iraq war – apart obviously from the dodgy dossier composed with the complicity of MI6? We went to war on the strength of information supplied by two ingenious fabricators. One of them, aptly named Curveball, was a fast-talking Iraqi refugee flying on the seat of his pants who, assiduously cultivated by his German keepers, provided us with Saddam's nonexistent mobile bio-labs – the same illusory vehicles that Colin Powell presented to the United Nations with much panache and the help of the CIA's colourful visual aids. Remember "slam-dunk"? That was the happy phrase with which George Tenet, at that time director of the CIA, personally verified the fabricated intelligence to his president, George Bush. Yet, when it came to the vote in parliament, what was being whispered to the doubters in the corridors? Let me guess: "If you'd seen the papers I've seen" – spoken with menace and conviction, and no doubt a hint of honest fear – "you'd know which door to go through!"

And I don't mind betting you, that's what was being whispered this time round. And that's the whole trouble about now. We're sliding back to where we started. We're either with them or we're with the terrorists. And of course, like other writers in the field, I have contributed to the spies' mythological status, even if my characters are divided about the things they do. And sometimes I feel a bit shifty about that. But I'm not alone. And I'm certainly not the first. And politicians are not dupes. Fifty years ago, I seem to remember, when covert action was the flavour of the hour, and politicians were being brought to the table to sign off on it, it was the pols as often as not, and not the professional spies, who bayed for blood.

"And those secret courts?" I hear you exclaim. "They're just ATMs for terrorists to sting Joe Public out of millions of pounds of compensation, for heaven's sake!"

But they're not. Joe Public is being stung all right – just not by the terrorists on this occasion, but by the people he pays to preserve his hard-won freedoms.

Copyright © David Cornwell June 2013