Labour grandee Jack Straw has disclosed his role in a 1970s Whitehall plot to examine private social security files belonging to the lover of the former Liberal leader Jeremy Thorpe.



The former foreign secretary said the “extraordinary” files, marked secret, went into intimate details about the relationship between Norman Scott and Thorpe. A memo written by Straw was forwarded to the then prime minister Harold Wilson.



Straw, then a special adviser, said Wilson had demanded that he go through the files to help Thorpe find out if there had been a political plot to prosecute Scott for social security fraud.



Wilson’s former aides disagree and claim that Straw was unwittingly being used by No 10 to gather evidence of Thorpe’s homosexuality that could be used to undermine the Liberal leader.



Most social security files are prosaic even by Whitehall standards, Straw said, but these were jaw-dropping.



“There were all sorts of pockets marked ‘secret’. We opened these pockets and there were statements made very many years before 1976, going into all sorts of details about the contact between Scott and Thorpe which went well beyond their use of social security numbers.



“They included graphic details ... and when and where they had had sexual encounters. It was very odd to think that all of this had been written down in longhand by a social security clerk in Barnstaple,” he told the Guardian.



Thorpe and three others were acquitted of conspiring to murder Scott in 1979 after one of the most famous court cases in British political history.



The case has been pushed back into the public sphere by the BBC’s recent TV dramatisation, starring Hugh Grant, called A Very English Scandal.



Straw’s involvement dates back to 1976 when he was a special adviser to Barbara Castle, the social security minister.

Thorpe was under pressure to resign from his position as Liberal leader following rumours that he was gay and about his alleged involvement in a plot to murder Scott.



Wilson had been approached by Thorpe with concerns that Scott faced a DHSS prosecution for a minor fraud as a way of airing claims of a gay affair, Straw said.



Straw, speaking to the media for the first time about his role in the affair, said he received a request from No 10 to find out why Scott had been prosecuted by the DHSS.

“Thorpe had contacted Wilson and claimed he was completely innocent of any wrongdoing and said that Scott’s prosecution for social security fraud for £14 and 60 pence had somehow been a put-up job.



“Wilson was predisposed to believe Thorpe, but asked us to look through the files,” he said.



Straw claimed the then prime minister was naive about sexuality. “Wilson had a boy scout’s attitude to sex – he really didn’t understand those urges,” Straw said.

The files were retrieved by the second permanent secretary, Sir Lance Errington, but their findings went well beyond the usual material.

Straw said they found evidence that Scott had for many years claimed Thorpe was his long-term lover.

“The documents seemed to back up Scott’s claims that Thorpe had withheld his national insurance cards,” he said.

Some reports later claimed that details of Scott’s records were tampered with while other details were leaked to the press. Straw denied any wrongdoing. “Wilson later appeared to imply that someone altered Scott’s national insurance records. This is categorically untrue,” he said.



Straw said the social security files also claimed that Scott had been in contact with the South African security services, BOSS, which had taken an interest in Thorpe.



“This was a time when the South African regime was trying to undermine anyone who had supported the anti-apartheid movement. So it was certainly of interest and passed on to the relevant authorities,” Straw said.



Thorpe died in 2014. Scott, who lives in Devon, claims there is still a cover-up of many aspects of the case.



This weekend, a suspect in the scandal came back into the frame when police returned to question him at his home. Andrew Newton, now 72, has been accused of hiring a hitman to kill Scott – before allegedly bungling an attempted assassination himself in 1975.



An investigation into the alleged plot was abandoned two years ago after police said Newton had died in May 2004. However, it emerged he is living in Dorking, Surrey, under the name Hann Redwin, forcing police to reopen the case. Two officers were seen arriving at Newton’s home on Sunday. They left after knocking on the door and receiving no answer.

•This article was amended on 5 June 2018 to correct the spelling of Barnstaple.