Late in 1920 the Hampstead branch of the Communist party of Great Britain held its end-of-year dinner at Pinoli’s restaurant in Soho, London. The diners were offered a range of dishes – fillet of whiting, chicken au chasserole, lamb cutlets, salads, ices and cheese – and the festivities ended, as Timothy Phillips writes, with a rousing rendition of the “Red Flag”. We know all this because Special Branch spies not only monitored the event but sent full details, including that insurrectionary menu, across Whitehall and even to members of the cabinet.

The Secret Twenties is the rewarding product of Phillips’s immersion in what he calls the “vast, sweeping, intriguing and sometimes shocking repository of material” that comprises the declassified archives of MI5. His subject is the intelligence services’ obsession with the new Soviet Union and the “Bolshevik threat”. While he is alert to the dangers of anachronism, and to how alarming the possibility of revolutionary contagion must have been, his evidence points to paranoia and error on the part of Britain’s spies, who in general were “simply scaring themselves with spectres of their own making”.

Journalist Francis Meynell smuggled Russian jewellery inside chocolates, licking the gems clean before flogging them

Along the way he makes a few significant discoveries and tells plenty of entertaining stories. The guest of honour at the dinner at Pinoli’s was Nikolai Klyshko, friend of HG Wells, who had not only brought two suitcases full of platinum bars into Britain to help fund the dissemination of Communist ideas, but instructed the left-wing journalist Francis Meynell to smuggle in expropriated Russian jewellery inside pats of butter and soft-centred chocolates. Meynell licked off the chocolate before flogging the gems in Hatton Garden.

While spies failed over years to identify the Comintern agent Jacob Kirchenstein, despite his rather conspicuous alias “Johnnie Walker”, they made much of the activities of Clare Sheridan, a cousin of Winston Churchill. Sheridan, a sculptor and socialite with advanced views on free love, had an affair with Soviet negotiator Lev Kamenev and paid a visit to the USSR, but while there was never a shred of evidence that she endangered national security she long remained one of the fledgling MI5’s most important suspects. Given her addiction to the limelight, Phillips reflects, “the idea that Sheridan could have functioned … as a secret agent is laughable”. There was “mania in the air”.

Facebook Twitter Pinterest Eisenstein’s 1925 film Battleship Potemkin was banned in Britain until the 1950s. Photograph: Ronald Grant Archive

In the same spirit of paranoia, British intelligence constantly surveilled the Society for Cultural Relations (vice-presidents John Maynard Keynes, EM Forster, Virginia Woolf et al) despite the fact that most of its meetings “discussed nothing more dangerous than contemporary Soviet theatre or Communist arts and crafts”. Sergei Eisenstein’s silent film Battleship Potemkin was banned in Britain until the 1950s. A Russian social club that “promised debauchery turned out to offer no more intriguing attractions than coffee and cards”, and thousands of personal ads in the press (“Swans are supposed to sing sweet songs of farewell – but not in Arcady – C”) were pored over by spooks on the lookout for espionage subtexts. In fact, as Phillips writes, the only 1920s “lonely heart” who definitely led a double life as a spy was the former Special Branch head Sir Basil Thomson, who was caught in Hyde Park having sex with a prostitute called Thelma de Lava.

In 1925, when the hard-right home secretary William Joynson-Hicks ordered a raid on the headquarters of the CPGB in Covent Garden, spy chiefs “were extremely surprised at the mildness and limited nature of what they seized”. Thanks to his years of research Phillips adds to our understanding of the raid in the same year on the London offices of the All-Russian Cooperative Society (Arcos). As the historian Richard Davenport-Hines has pointed out, the identification in The Secret Twenties of MI5’s Arcos informant as Edward Langston, afterwards a pub landlord, counts as a “major scoop”. No evidence of subversion was found in the raid, though diplomatic relations with the USSR were broken off to cover up the government’s embarrassment. It was, all in all, an intelligence calamity: having now been alerted to British spies’ ability to read their telegrams, the Soviets changed their operations and no high-grade communications were decrypted again until 1945.

Thousands of personal ads in the press were pored over by spooks on the lookout for espionage subtexts

It might be argued, in response to Phillips’s notion of “hysteria” and a “vicious cycle of suspicion”, that British intelligence was merely doing what it was supposed to do, and that Moscow did indeed have plans for, say, industrial sabotage in Britain. But the book also offers a reminder of rightwing eagerness to identify the whole of the British labour movement with Bolshevism. As the Labour party became more popular, spies forged links with rightist journalists and Tory MPs, to challenge policies that were not in the least insurrectionary but did mean “a weakening of the power and wealth of Britain’s elites”. At the same time as the Labour leader Ramsay MacDonald identified the Russian Revolution as having been “marked by a ruthless dictatorship of cruel repression”, so-called “Diehard” Tories “took hope from the increasing strength of popular reactionary and even proto-fascist groups”.

The notorious Zinoviev Letter that triggered the fall of the first Labour government in 1924 was a forgery leaked by British intelligence to officers of the Conservative party, who passed it on to the Daily Mail. The 1999 official report into the scandal noted that the security and spook community at the time consisted of a “very incestuous circle, an elite network”, who went to school together; their allegiances “lay firmly in the Conservative camp”. Phillips identifies the Letter as the occasion when “a ‘deep state’ first made its presence in modern Britain, the moment when a concerted and thoroughly illegitimate attempt was made to subvert a national democratic process”. His book is too spirited and nuanced to hammer home an oversimple argument about the establishment and surveillance, but Phillips was no doubt interested in the recent news that an entire file on the Zinoviev Letter was accidentally and mysteriously “lost” by Home Office civil servants.