When the US government contractor Edward Snowden disclosed the extent of the global eavesdropping operation being run by the National Security Agency, GCHQ and their partners in Australia, Canada and New Zealand, many people spoke, aghast, of the way in which the 21st century appeared to have ushered in a new age of surveillance.

Yet spying on a mass scale has of course been with us for as long as governments have been skittish about foreign powers or uncertain about the loyalties of their own citizens. When Oliver Cromwell established the Post Office as a state monopoly in 1657, he did so not because he wished to improve England’s communications, but because the opening of mail was considered to be “the best means to discover and prevent any dangerous and wicked designs against the Commonwealth”.

In 1703, a year into Queen Anne’s reign, an Oxford don, Edward Willes, was appointed head of the new Post Office “deciphering branch”, which not only opened letters but also broke correspondents’ codes. One of the few people who twigged was Jonathan Swift, who wrote in Gulliver’s Travels of a government department that opens letters so that it “can discover a close-stool” – a toilet – “to signify a privy-council; a flock of geese, a senate; a lame dog, an invader”.

The rest of the country was kept in the dark for more than 140 years. By the time the existence of the deciphering branch was made public by some hyper-vigilant Italian revolutionaries living in exile in London, the Willes family had turned the opening of letters, cracking of codes and unmasking of supposed plots into a dynastic enterprise: the department was being run by Willes’s grandson, Francis.

One of the key points made in Rhodri Jeffreys-Jones’s book is that surveillance in the UK and US is frequently a private enterprise rather than a state operation, and that in terms of the subsequent harm done to individuals, private surveillance routinely outperforms its public counterpart.

When Allan Pinkerton left his home in the Gorbals district of Glasgow in 1842, he claimed to be one step ahead of the police, who were pursuing him because of his activities for the Radicals, the Scottish equivalent of the Chartists. That may or may not have been true – Jeffreys-Jones raises the possibility that he may have been a police spy, and was fleeing from his fellow activists. What is not disputed is that the Pinkerton National Detective Agency, which he founded in Chicago, specialised in the infiltration of trade unions, and contributed to the breaking of one strike after another during the late 19th and early 20th centuries.

Workers on both sides of the Atlantic continued to be surveilled by their employers, who would take an extraordinary interest in the amount of time they spent in the lavatory, the telephone calls they made, and any complaints they had lodged about unsafe working conditions. In the UK, Admiral Reginald “Blinker” Hall, the head of GCHQ’s first world war precursor, 40 OB (Old Building), became so alarmed at what he saw as the twin threats of Bolshevism in Russia and an increasingly strident trade union movement at home that in 1919 he founded an organisation that would compile lists of potential troublemakers.

The Economic League eventually blacklisted tens of thousands of workers, and sold access to its database to many of the household names in British banking, construction, retail, pharmaceuticals and motor manufacturing. It was a curiously resilient organisation, surviving eight major exposes in the press and on television between 1937 and 1988. When it finally closed down in 1993, it was simply reborn as the Consulting Association, whose blacklist continued to be available to prying employers for a further 16 years.

By the time its operations ceased for good, it had ruined the lives of countless safety campaigners, trade unionists, and their families, and workers had become so closely monitored that sociologists were describing them as “transparent”. One union activist had another way of summarising the human consequences of blacklisting: “Missus goes, house goes, then suicide.”

Jeffreys-Jones’s rigorous and highly readable history of surveillance on both sides of the Atlantic is far from presenting state, as opposed to private, surveillance as being entirely benign; all too often, he writes, “the silhouette of the state can still be discerned behind some private malpractices”. Furthermore, he concludes that “where governments possess surveillance powers they will, eventually, abuse them”. One of his most compelling chapters, entitled McCarthyism in Britain, details the way in which the Cabinet Committee on Subversive Activities, established in 1947 under pressure from the US, vetted about 10,000 civil servants, with 124 being removed from their posts. It was, he concludes, a vindictive episode, but one that passed largely unnoticed at the time because of the unobtrusive manner in which MI5 went about its work.

No examination of more modern abuses of surveillance powers by the British state would be complete without recounting the way in which undercover Scotland Yard detectives formed relationships with some of the political activists they were spying on between the 1980s and 2010, and conceived children with a number of them, before disappearing from the lives of both women and children once their espionage missions were concluded.

But herein lies the problem for those people who believe state surveillance should be forever exposed and challenged unless it can be shown to be directed at terrorists and criminals: the “Snowdenites”, as Jeffreys-Jones calls them. While such transgressions as the undercover cops’ operations diminished the Yard in the eyes of some members of the public, the overwhelming majority of people in the UK said they still trusted the police.

Moreover, in the UK at least, there is underlying support for state surveillance. This is not because members of the public have considered, and accepted, the securicrats’ argument that there is a distinction between “bulk capture” of data and mass surveillance. In 2014, several months after Snowden’s first disclosures, more than 70% of people polled said they believed the UK government should prioritise the threat posed by terrorists and serious criminals, “even if this erodes people’s rights to privacy”.

We continue to learn more about how these competing interests are weighed: just last month the latest WikiLeaks data dump revealed that the US and UK intelligence services had conspired to turn TVs into eavesdropping devices. That leak prompted Michael Hayden, a former director of the CIA, to warn that the agencies “may be running into this different cultural approach”, of whistleblowing, as a consequence of their being obliged to hire from a new demographic cohort. “I don’t mean to judge them at all,” said Hayden, “but this group of millennials … simply have different understandings of the words loyalty, secrecy and transparency than certainly my generation did.” There may be more Snowden-style troubles ahead.

• Ian Cobain’s The History Thieves: Secrets, Lies and the Shaping of a Modern Nation is published by Portobello. To order We Know All About You for £14.24 go to bookshop.theguardian.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99.