When I entered government in 2010, I was the leader of a party that had been out of power for 65 years. There were a lot of things we had to re-learn, and a lot that was surprising and new.

When a senior official took me aside and told me that the previous government had granted MI5 direct access to records of millions of phone calls made in the UK – a capability only a tiny handful of senior cabinet ministers knew about – I was astonished that such a powerful capability had not been declared either to the public or to parliament and insisted that its necessity should be reviewed.

That the existence of this previously top secret database was finally revealed in parliament by the home secretary on Wednesday, as part of a comprehensive new investigatory powers bill covering many other previously secret intelligence capabilities, speaks volumes about how far we’ve come in a few short years.

The securocrats in the Home Office had to persuade two sets of ministers from different political traditions

When the Edward Snowden revelations were first published, the knee-jerk response from the government was to play the man and ignore the ball. The debate online and in the pages of the Guardian was taking off, and civil society groups were beginning to mount legal challenges in the investigatory powers tribunal. But most ministers simply didn’t understand – whatever concerns they may have had about Snowden’s own behaviour – the significance of the fact the world now knew the government’s most closely guarded secrets. They refused to acknowledge that the democratisation of the security state had become inevitable.

Meanwhile, the Home Office hadn’t deviated from its robotic approach to demanding ever greater powers. It had presented successive governments with near-identical proposals for broad new internet surveillance powers – the so-called “snooper’s charter” – without sufficient thought for our privacy and without any serious effort to justify them on the basis of operational need.

Only 'tiny handful' of ministers knew of mass surveillance, Clegg reveals Read more

One of the merits of coalition politics is its incompatibility with sofa government. For once, the securocrats in the Home Office had to persuade two sets of ministers from different political traditions. In the case of my party, our inbuilt mistrust of big government and belief in civil liberties meant we took the time to scrutinise the proposals that were put to us in huge detail. While we certainly developed great respect for the dedication and professionalism of the intelligence agencies, we didn’t sign off everything that arrived on Home Office notepaper.

This combination of internal friction and external public debate proved to be a potent recipe for change. We overcame the head-in-the-sand attitude of securocrats and Conservative ministers, first by vetoing the draft communications data bill in 2013 (to dire Home Office warnings of blood on the streets), and then by attaching conditions to every new piece of legislation that dealt with security matters.

That allowed us to set up a root-and-branch review of the existing surveillance laws, which was conducted by David Anderson QC; to convene a civil society panel under the auspices of the Royal United Services Institute; to appoint a senior diplomat to wrest control of the international data-sharing agenda away from the Home Office; and to place a “sell-by date” on a substantial chunk of the powers, forcing the government elected in 2015 to bring forward fresh, comprehensive legislation by December 2016.

The draft bill that was published this week is the result. It is far from perfect. Many of the powers it contains are controversial and have evolved over time without any meaningful parliamentary scrutiny. The ability, for example, of GCHQ to hack anything from handsets to whole networks is highly intrusive and needs to be much better understood before we can place it within appropriate constraints.

Theresa May’s investigatory powers bill is a step in the right direction | Keir Starmer Read more

The new, revised proposals on the storage of web browsing data remain problematic as the bill appears to call for the storage of vast quantities of data that goes far beyond the operational requirements set out by the home secretary in the Commons. The so called “double lock” of judicial oversight appears to be nothing of the sort, as judges will have very little discretion when making decisions about individual warrants. And many will question the access the intelligence agencies have to our phone records.



But pause and compare this bill to every recent attempt to legislate in this area and we see evidence of a remarkable journey. A story of successfully mobilising people outside the traditional circles of advisers and ministers to force the pace of reform, leading to an opening up of what was a highly secretive business.

We are now in touching distance of a comprehensive new law covering every single surveillance power at the disposal of the government – which will have been thoroughly debated in both houses of parliament, including scrutiny of hitherto secret capabilities. Whatever people may think about the decisions eventually made, democratic oversight will have been strengthened, transparency enhanced, and secret powers forced out into the open.