The establishment’s tendency to cover up its own scandals is on full display in the latest releases from Whitehall’s cupboard of secret “too hot to handle” files.

The “top secret and strictly personal” files, known as the cabinet secretary’s miscellaneous papers, date back to 1936 and include material on double agents Kim Philby and John Cairncross, the Profumo affair and corruption concerns over a future Tory home secretary, Reginald Maudling.

They were kept unregistered and uncatalogued under lock and key in the Cabinet Office until their existence was officially acknowledged for the first time in 2015.

The files show that it was believed that “at least 40” ministers or MPs were patients or had met Stephen Ward, the osteopath and “fixer” with strong links to the Soviet embassy who was at the centre of the 1960s Profumo sex and spy scandal.

They also confirm that Philby had been invited to dinner at the British embassy in Beirut on the night of his disappearance after he had confessed to being a Soviet spy and that it was decided not to disclose that Cairncross had been a Soviet spy because his brother was chief economic adviser to the government at the time.

The Profumo file shows that the cabinet secretary and MI5 had serious concerns about Ward’s links with senior political and Whitehall figures in autumn 1961, two years before the scandal was made public.

The file includes a note of the cabinet secretary’s warning in August that year to the then defence secretary, John Profumo, of Ward’s friendship with Yevgevny Ivanov, a Soviet spy attached to the Russian embassy. Profumo confirmed he had met Ivanov at Lord Astor’s Cliveden estate where Ward had a cottage and use of the swimming pool. As a result of the warning Profumo broke off his affair with Christine Keeler, to whom he had been introduced by Ward.

“Mr Profumo told me that Ward, in his practice as an osteopath, sees a number of people who are prominent in public life, including one or two members of the present government,” cabinet secretary at the time, Norman Brook, recorded.

He spoke to one of them, the chancellor of exchequer, Selwyn Lloyd, who told him that Ward was also an artist and had recently done a drawing of him. “[Lloyd] agrees that Ward is in a position to chatter with a number of notabilities – and that it would be be a good thing if we could put people on warning about him.”

In November 1961 a government whip who had met Ward at a Soviet embassy party where “the latter seemed very much at home” reported “he knew, or believed, that his patients included at least 40 people who were ministers or members of parliament”. It was not until the scandal became public in 1963 that anything much seems to have happened as a result.

Even then the home secretary, Rab Butler, decided “it was not necessary to institute any special inquiries at the moment” into “the security implications” of Ward’s possible links with MPs and ministers, but “if he hears of any parliamentary contacts he will let us know”, recorded the cabinet secretary. Instead senior civil servants of the rank of undersecretary and above – as it “would not be worthwhile unless they were pretty high-standing”– were all asked whether they had ever come across Ward. Unsurprisingly, none told the cabinet secretary they had.

The Philby file, which documents the flight to Moscow of the “third man” in the Cambridge spy ring on 23 January 1963, confirms a similar unwillingness to blame those in high places. Philby had been working as a journalist in Beirut since he was officially cleared in 1955 by the then foreign secretary, Harold Macmillan.

It was not until 1 July 1963 that his defection was officially confirmed. A “roundup on Philby” note for the cabinet secretary two days laterreads: “One of the more difficult points looks like being … the staff of HM embassy at Beirut should have been warned not to have anything to do with Philby immediately after his admission that he had been a spy, whereas in fact the first secretary invited him to dinner the night he disappeared.

“HM ambassador was informed personally about the admission but probably did not tell the rest of the staff. We do not want to blame him for this. The best comment probably is that … more harm than good would have been done if the British officials had suddenly and demonstratively changed their attitude to Philby.”

A similar desire to protect those in high places was also evident in the Cairncross case. In March 1964, Cairncross, who was by then was working as an academic in the US, confessed to having spied for Russia between 1936 and 1951, during which time he had worked for the Foreign Office, Treasury, GCHQ, MI6 and the Ministry of Supply.

After failing to secure the cooperation of the FBI in returning Cairncross to stand trial because they did not consider him a current security risk,Macmillan, then prime minister, agreed that the case should not be made public because it could lead to criticism of the Americans, but also because “it would be bound to call into question the position of Cairncross’s brother”.

The then cabinet secretary, Burke Trend, said that quite apart from the “distress and embarrassment” to Cairncross’s economist brother, Alec, “we have to ask ourselves what would be the probable result, in terms of public policy in the widest sense, if it became known that the government were employing, as their chief economic adviser, a man who was the brother of a self-confessed Communist spy”. As a result, no further action was taken in 1964 and it was not until 1990 thatJohn Cairncross was publicly confirmed as the “fifth man” in the Cambridge spy ring.

As far as Maudling was concerned, his “too hot to handle” file shows concerns were raised about his probity 11 years before he resigned as home secretary amid the Poulson corruption scandal. Lord St Aldwyn, then the Tory chief whip in the House of Lords, had reported that “he had recently heard two ministers discussing, at the Carlton club, the readiness of a colleague [Maudling] to accept presents from people with whom he had official contacts”.

The cabinet secretary notes: “I have reason to fear that Mr Maudling may be a little to ready to accept presents” and cites examples of gifts of cine-cameras, wristwatches and a “suit length from a Yorkshire firm”. However, they decided not to do anything about “Mr X”, as some of the official papers refer to him, comforting themselves that his move from president of the board of trade to the Colonial Office meant the presents from its “less sophisticated clients” were likely to prove less embarrassing.