"Freedom of Information Act. Three harmless words. I look at those words as I write them, and feel like shaking my head 'til it drops off. You idiot. You naive, foolish, irresponsible nincompoop. There is really no description of stupidity, no matter how vivid, that is adequate. I quake at the imbecility of it."

Thus Tony Blair records in his memoirs what he believes to have been one of his greatest mistakes while in office: introducing legislation intended to shed light on government in a manner that empowered people.

"It is a dangerous act," he went on, because governments need to be able to debate and decide issues in confidence.

In opposition, Blair had said he believed such an act would "signal a new relationship between government and people: a relationship which sees the public as legitimate stakeholders in the running of the country". As prime minister, he saw it as a law that was "utterly undermining of sensible government".

The allegations facing Michael Gove and his special advisers suggest that within days of the general election they were using personal email accounts in a way that prevented FoI scrutiny.

But the reluctance of some in government to comply with the act became apparent even before its introduction in January 2005, almost 40 years after a similar law was passed in the US. The Cabinet Office told staff to destroy millions of emails ahead of the act, while other government departments doubled the number of files they were shredding.

There seems little doubt that the material destroyed would have contained information of considerable public interest.

The act has been employed to reveal the level of farming subsidies paid to the UK's wealthiest landowners, the numbers of black and minority ethnic students admitted to Oxbridge colleges, and NHS "postcode lotteries" for costly medication.

The full details of MPs' expenses claims would not have been acquired by the Daily Telegraph had the parliamentary authorities not been preparing a heavily redacted document for FoI release.

According to a report last year from Robert Hazell (pdf), a former civil servant who is now a politics professor at University College London, and Dr Ben Worthy, the FoI act has not undermined the ability of civil servants to give frank advice, as Blair claims, nor affected government record-keeping.

But they concluded that while the act has achieved its core objectives of greater transparency and accountability, it has done nothing to achieve three of its four secondary objectives (improved decision-making and better public understanding and participation in government) and has hindered progress towards its fourth (increased trust).

They said most news reports based on information obtained through FoI had the effect of reducing trust. "This is because of the media's predominantly negative reporting, exacerbated by government resistance to media requests, and pre-existing low levels of trust."

Maurice Frankel, the director of the UK Campaign for Freedom of Information, believes the act is working as intended, and that the greatest improvement in recent years has been in the performance of the information commissioner's office (ICO), which upholds the act and handles complaints that information is not being disclosed. Complaints to the ICO which had taken up to four years to resolve and now being handled in as little as six months, he says.

The ICO also monitors the speed with which public bodies respond to FoI requests, focusing on those that fail to answer in 20 days as the law requires.

Most of the repeat offenders are local authorities, although a recent ICO monitoring exercise also examined the performance of two police forces – Surrey and City of London – and the Equality and Human Rights Commission. Only one central government department was included in the exercise: the Department for Education.