Keith Jeffery, author of the first – and possibly only – official history of MI6, said today he had made a "Faustian pact" that had in some cases "overridden the imperatives of historical scholarship". But he was given an offer he could not refuse – "the holy grail of the British archives".

Those archives are records of MI6 operations from 1909, when Britain's Secret Intelligence Service was set up, to 1949, when the history stops. MI6 said today that its archives, unlike those of the domestic Security Service, MI5, and the wartime Special Operations Executive (SOE) would remain closed to the public. "As a historian I regret they won't release the documents," Jeffery, professor of British history at Queen's University Belfast, said.

On grounds of national security, he was asked not to include some material, notably the names of MI6 officers, even though they had previously been identified in what he describes as reliable and scholarly works. Immense quantities of MI6 documents were destroyed, mainly because of lack of space in its offices, Jeffery said. "I have found no evidence that destruction was carried out casually or maliciously, as some sort of cover-up," he writes in the book's preface.

"I am not aware of a single case where the reason [why] it was necessary to destroy was embarrassment," said Sir John Scarlett, former chief of MI6, who gave the green light for today's publication. However, there is no mention of deeply embarrassing episodes, for example of Kim Philby, the Soviet agent inside MI6, described by Jeffery as "part of the charmed circle". Philby is widely believed to have tipped off Moscow in 1945 about Konstantin Volkov, a Russian intelligence officer who planned to defect and spill the beans on Soviet agents inside MI6. The official history stops at 1949, when MI6 began to expand as the cold war took off. There is no mention of Philby's alleged role in the betrayal of Albanian exiles who planned to overthrow the government there in 1949.

"With Philby, it goes kind of well, and then there is a dot, dot, dot," Jeffery said, referring to the cut-off date. He said there was almost nothing in the MI6 archives about the persecution of the Jews and the Final Solution. However, the files do show that Whitehall refused to circulate a report from an MI6 agent warning of the 1939 Nazi-Soviet pact and that MI6 warned of Hitler's true ambitions at the time of the 1938 Munich crisis – advice that was unwelcome to appeasers in Whitehall.

The book also reveals how MI6 faked the death of Horst Kopkow, a German expert in Soviet intelligence suspected of war crimes, "in order to facilitiate [his] future use". Jeffery said: "I looked very hard for 'bad stuff'. In the end, I found less evidence than perhaps we might have expected, certainly less evidence than I might have expected as the amateur espionage fiction buff that I was." If there had been none in the book, no one would have believed it, he said earlier this week.