UK security service went back and forth for decades over whether the communist Paddy Costello was a Soviet agent

A controversy that has spanned more than seven decades over whether a New Zealand-born Cambridge-educated diplomat and academic was a top Soviet spy has been fuelled by the release of his MI5 file.

Desmond Patrick Costello was named by the official MI5 historian, Christopher Andrew, in 1999 in the Mitrokhin Archives as one of the KGB’s most important spies in the 1950s when he served as a leading New Zealand diplomat in Paris.

But it has long been argued that “Paddy” Costello, as he was known, was a victim of a British MI5 “whispering campaign”. A recent biography by Sir James McNeish, The Sixth Man, made the case for him as a gifted New Zealander who was attacked by a generation of spyhunters.

The MI5 files released at the National Archives in Kew on Tuesday fail to provide a conclusive answer but show that MI5 was at one stage locked in a decade-long battle with the New Zealand government over whether Costello should be dismissed as a diplomat. During this struggle, the security services were forced to concede that their case against Costello was “a thin one”.

Dr Richard Dunley says in a National Archives blog that the newly released files show that Costello first came to the attention of MI5 when he moved to Britain to study on a scholarship at Cambridge University in the mid-1930s, when the Cambridge spy ring was being established.

But as Dunley notes, Costello – unlike Kim Philby or Guy Burgess – openly espoused his new-found communism, and developed links with leading figures in the party including James Klugmann.

He became a classics and modern languages lecturer at University College of the South West, Exeter, but was forced to resign because of his open communist views in 1940 when a student at the college was convicted under the Official Secrets Act, although Costello had no direct involvement.

He joined the army and served as an intelligence officer and translator for the New Zealand forces in north Africa and Italy before being appointed by the New Zealand government in 1944 as second secretary to their legation in Moscow.

Costello had reputedly told the New Zealand prime minister he was “a little bit left-wing” only to be told: “Oh well, it won’t hurt us to have one or two communists in Moscow.”

But while the New Zealand Department of External Affairs was very pleased with its recruit, describing him as “one of the best people the New Zealand government have”, MI5 officers were horrified that a man they suspected of having links with Soviet intelligence now had access to confidential official information.

A security service protest to the New Zealand high commission and the Dominions Office in London only brought the response that Costello was not seen as a risk. Sir Eric Machtig, of the Dominions Office, said the evidence only pointed to “a familiar form of infantile communism which most people grow out of”.

The file shows Roger Hollis, head of MI5, arguing in 1944: “You may think the case against Costello himself is a thin one, and I think I should add that we have information from entirely reliable but very secret sources that certain of the Communist party leaders were aware of Costello’s departure from this country in July last.”

Costello continued his career in the New Zealand diplomatic service and in 1950 was promoted to be first secretary at the Paris legation. Dunley says this led to a four-year battle between MI5 and the New Zealand authorities over whether he should be removed.

It was his appointment to the Paris job that also led to Andrew’s claims that he was one of the most important Soviet spies at the time and subsequent false claims that he provided New Zealand passports to the atomic spies Peter and Helen Kroger.

Facebook Twitter Pinterest Bella Costello, his wife. Photograph: Crown copyright/National Archives

Although the New Zealand authorities defended the integrity of their diplomat, by 1955 the British – joined by the Americans – made clear that his continuing employment was putting their willingness to share intelligence at risk. He was forced out and returned to Britain where he took up the chair in Slavonic Studies at Manchester University. There, with his wife Bella, he entertained a constant flow of Soviet guests. Dunley says the file shows that while his politics and links with parts of the Soviet government were clear, MI5 struggled to prove any involvement in Russian intelligence work.

However, in December 1960 the file shows that the security service identified Bella as a Russian intelligence service agent because of her role in an espionage operation aimed at acquiring death certificates of long-deceased British children to build false identities for Russian agents. “It seems highly probable that both Costellos are agents of the RIS, Costello’s role may well be that of a talent spotter,” notes the MI5 file.

Dunley says MI5 “finally felt they had their man” in June 1963 when Costello, then under close surveillance, was seen to meet two known Soviet intelligence officers. A further meeting took place in November 1963, but the details of the meetings are sketchy and there were doubts about the identity of the alleged Russian spies. The file ends just before Costello’s death in 1964 without shedding any further light on his alleged Soviet connection. His son, Mick Costello, did become a leading figure in the Communist party of Great Britain.

Dunley says that throughout much of Costello’s life, MI5 went back and forth over whether he was a Soviet agent or simply a politically active intellectual: “The debate has continued in the open since his death and the documents released today will provide a fascinating new insight into the man, but they are unlikely to entirely satisfy either side. This story appears to be one with more to run.”