"You idiot. You naive, foolish, irresponsible nincompoop," Blair said to himself in his memoirs.

"There is really no description of stupidity, no matter how vivid, that is adequate. I quake at the imbecility of it."

Was he discussing the deaths of hundreds of thousands in Iraq? The dismantling of Britain's civil liberties? His decision to let John Prescott stand in for him at PMQs? No. Of all Blair's decisions, it was the Freedom of Information (FoI) Act which sparked a crisis of self-doubt.

It's a doubt the civil service has done its best to address. Since the Act was passed it has become increasingly useless. Now, after four years of coalition government, the FoI Act is barely worth the paper it's written on. One of the most powerful pieces of transparency legislation this country produced has been neutered.

In recent months, the Department of Work and Pensions (DWP) has fought a fierce legal battle to prevent the publication of the risk register into Iain Duncan Smith's Universal Credit scheme, along with other documents into concerns and problems with the plan. The information commissioner ruled it should be released, minus the risk register. A tribunal agreed, but wanted the risk register published too.

The DWP fought the case tooth-and-nail but lost. It appealed and lost that too, this time with a venomous response from a judge who could see "no support for the argument" and pointed out that the department had not "provided any persuasive evidence". The DWP asked for permission to appeal again and has been granted the opportunity to do so at an oral hearing in a few months.

How much is all this costing, you might wonder? Well wonder all you like, because you'll never find out.

My FoI to the DWP asking for costs came back recently with a flat refusal, as expected. "The department does not keep a record of the time its staff spend on particular Freedom of Information case work so the information you seek is not held," it said. The full cost to the public purse would have involved the costs of the DWP, the Treasury Solicitors Department and the Information Commissioner's Office, but even this small piece of the puzzle was considered confidential.

These legal cases are not small fry. The grade of barrister involved in them usually cost £3,000-plus a day and countless government appeals mean they go on for months, even when the judge's disparaging remarks have already made clear they have almost no chance of success.

It’s the same across government departments. We know from the scraps of information in Home Office accounts that it haemorrhages cash on Theresa May's constant legal battles and appeals. These included £8.9 million on liability for unlawful detention by the UK Border Agency (UKBA), £37.5 million in a dispute between UKBA and former asylum accommodation providers and £21.2 million for outstanding cases in which UKBA changes to immigration rules had a negative impact on institutions.

This is just the tip of the iceberg, but when campaigning group Brit Cits tried to find out more, the Home Office - without a trace of irony - told her it would be "disproportionately expensive" to provide the information.

What goes for FoI goes for parliamentary questions as well.

Shadow justice secretary Sadiq Khan recently asked a series of basic parliamentary questions to then-prisons minister Jeremy Wright. He was told it would be too expensive to "manually interrogate" prison incidence reports to find out how much contraband was getting into jails and how. In a string of questions, Wright basically just stamped the words 'disproportionately expensive' across each one.

When Tory MP Philip Davies asked which religions were being targeted in religiously-aggravated attacks, he was told that "data on religious belief of offenders or victims are not reported centrally to the Ministry of Justice".

There was a similar response when campaigners from the Howard League used FoI to ask how many times their name had cropped up in internal correspondence. They have a sneaking and not-unjustified suspicion that the justice secretary does not like them very much and may be making policy decisions to spite them. "I am afraid that I am not able to confirm whether the Ministry of Justice (MoJ) holds the information you have requested," the department wrote back. "On this occasion, the cost of determining whether we hold the information would exceed the limit set by the Freedom of Information Act and, as a result, I am afraid will not be taking your request further."

In its response the department let slip that it can only search for the titles of emails, not the content of the message.

It is symptomatic of the way governments have responded to FoI: not by embracing the spirit of openness but by finding ways of avoiding compliance while staying within the letter of the law.

The first response is not to record information at all, as in my FoI request. Alternately they just don't to hold it centrally, as in the MoJ's refusal to collate incidence reports from prisons about contraband. They can then cite an S12 cost of compliance exemption, saying it would cost more than £600 to collect the data.

Or they cripple themselves with ineffective systems, like email searches than can only scan titles rather than the message content.

This approach is hopeless if one is interested in effective government. What sort of company would make do without information as pertinent as that which the MoJ claims to have no interest in? If the government cares about contraband it may wish to find out how it gets into jails. If it cares about hate crime it may wish to find out which faiths are being targeted.

One quick glance at Google Analytics gives me much more information about the aims and achievements of this website than government departments have about matters of national importance. A Google Mail account allows far greater search and retrieval than the system employed in Whitehall. We're not talking top-level data here. This is simple stuff.

When these tried and tested methods fail, departments cite an S22 exemption, saying the information is intended for future publication, although they can rarely confirm when the future publication will take place.

Or they cite S35 on the formulation of government policy – an exemption intended to relate to high-level government police but which is increasingly applied far more broadly.

Or – as in the DWP risk register case – they cite the S36 exemption on the basis of prejudice to effective conduct of public affairs. As several judges have made clear, this is overwhelmingly used to save ministerial blushes rather than protest honest internal debate.

Finally, they cite the S43 exemption on commercial interests to keep the activities of privately-run public services, such as those delivered by Atos and G4S, out the public eye. Labour has pledged to make these firms subject to FoI if it gets to power but even if it does so, the commercial interest exemption allows the department and the contractor to play a game of hot potato, constantly citing commercial sensitivity in refusing to reveal information which is manifestly in the public interest.

(Incidentally, the mad effects of privatisation are more far-reaching and pernicious than that. Justice minister Simon Hughes recently revealed he couldn't visit a women's rehabilitation centre, because to visit one and not all of them would potentially open up the MoJ to judicial review after the work is contracted out. A child who had just drunk five pints of Coca Cola could not have developed a more problematic system.)

The reason Blair felt so strongly opposed to FoI was because it was such a genuinely good piece of legislation: transparent, far-reaching and closing the gap between the state and the public. But it wasn't enough to pass it. Laws like that need to be tended to make sure they are achieving their desired objective. Instead, the political class has looked aghast at what it did. Civil servants and ministers have taken notice of that and stacked the odds against publication.

Their method for doing so has significantly constricted the potential for effective government. It is a telling indication of their priorities: faced with a choice between being able to do their jobs properly and limiting embarrassment, they opted for the latter.

It is part of a depressing tradition in Westminster. When a genuinely radical innovation is introduced, the system shuts it down. That's why Sarah Wollaston – the independent-minded MP selected by an open primary for the Tories – was the last of her kind. The scheme was not cancelled because it didn't work. It was cancelled because it did work. The same thing is happening to Freedom of Information.