The frightened and frightening myopia of the Republican party has once more damaged American interests and American democracy. While still a minority in the Senate – as they will not be next year – the Republicans managed to muster a blocking 42 votes to kill a bill that would have brought some aspects of the NSA bulk surveillance programme under greater democratic control. They did so using rhetoric that would make even the shade of Richard Nixon blush. “God forbid we wake up tomorrow and [Islamic State] is in the United States,” said one.

This defeat was understood by some of the NSA’s more thoughtful defenders to be a horrendous own goal. First, the concessions in the bill were not very great. The most interesting one would have put a privacy advocate on the secret court that hears secret applications to authorise particular operations. That way the arguments for civil liberties could no longer go unheard: they would have to be more actively ignored. The companies whose data was a target for surveillance would know more about that monitoring. But the provisions limiting who might be watched and why remained extremely vague, to the disappointment of all defenders of civil liberties.

The apparent concession that the NSA would no longer hold years and years’ worth of communications data is worth very little in practice, since the telecommunications companies would still hold on to it. As we know from experience in this country, it is not very important who holds that data if the security services can access it whenever they want and combine it in ad hoc ways to build the kind of database panopticon they want. What is more, the definition of a legitimate target expanded in earlier stages of the bill, turning it into an amorphous, greedy phrase that might mean anything: “a specific selection target” might mean a person, or a company, possibly even an area code.

So this was not a bill that could command the wholehearted support of anyone who cares about civil liberties. But it represented at least an attempt to draw up a principled and comprehensible framework within which the intelligence and security services could do their vital work in the digital world. Now that it has failed we are back in the business of messy, ad hoc and incoherent compromise. The courts will work in one direction, and the politicians in another. That is true on both sides of the Atlantic. It cannot and should not last. In June next year the truly obnoxious section 215 of the Bush-era Patriot Act will expire. This became the mechanism used to legitimise the NSA’s mass harvesting of metadata on phone calls. No one now knows whether it will be renewed by the new Congress, in which there will be a Republican majority but one divided over whether Americans should be more frightened of foreigners or of their own government.

In this country, too, legislation governing electronic surveillance is both outdated and incoherent. Parliament and Congress have not yet shown they are fit stewards of our conflicting interests, nor even that they fully understand the complex arguments involved in balancing liberty and security.