It was 1997, the 50th anniversary of the suspected flying saucer crash at Roswell in New Mexico, and the heyday of the paranormal mystery series The X-Files. The English-speaking world was gripped by UFO-mania. But what seemed a delightful mystery to some was becoming a headache for the spooks at Britain’s Defence Intelligence Staff.

Analysts at the DI55 office, the department lumbered with the UFO brief, were being peppered with requests from ufologists – and even parliamentary questions – for information on flying saucers, taking up time they felt would be better spent on terrestrial defence matters. So top brass decided to undertake a definitive study of the unit’s collection of reported UFO sightings to establish, once and for all, whether there was anything in them.

Previously unseen documents reveal that, far from being an objective study into the possibility of extraterrestrial visitors, the report was intended from the start to absolve the Ministry of Defence of responsibility for investigating sightings. Messages between officials at DIS and the contractor carrying out the research show that it focused from the outset only “on the possible threat to the UK [from hostile foreign powers] and technology acquisition” and not “X-Files activities such as alien abductions”. A separate memo says: “It shouldn’t be driven by a UFO thesis.”

The study replaced one mystery with another after its author determined that the UFO sightings were a result of unexplained plasma formations in the atmosphere. Nevertheless, DI55 announced afterwards that it would no longer accept UFO reports.

The documents show the deliberations behind the research, which began in 1997 and collated the previous decade’s worth of UFO sightings – known in the technical jargon of DIS as unidentified aerial phenomena (UAPs) – in a database. The identities of all the officers involved in the conversation, which lasted several years, are redacted. Many other excerpts are blacked out, with exemptions cited including risks to national security and international relations.

“The increasing media attention given to this subject in recent months has almost doubled the work of the desk officers involved to the detriment of other tasks more directly relevant to the work of the branch,” one memo says, adding that it was now time to “reappraise the situation” and clarify DIS’s role in the issue.

Officials complain about the cost in time and money of investigating sightings and responding to questions. “The problem is unlikely to subside, especially as the US brings into service over the next decade high flying capabilities such as Global Star, Dark Star, the X-33 and, should it come to fruition, the manned spaceplane,” writes one senior officer. “Other nations will follow, especially with UAVs [unmanned aerial vehicles], which may permit risk-taking in unauthorised penetrations of airspace.”

Other correspondents remained open-minded, with one arguing that in the absence of analysis “we can not discount the possibility that UAPs are real” or that they could be natural phenomena but with “possible military applications”. “It could be argued that UAPs pose a potential threat to the defence of the realm since we have no idea what they are,” the writer continues.

It emerges that the analyst commissioned to write the report had a personal interest in the subject. Pointing out that the same UFO questionnaire had been in use since the 1950s, he notes: “I filled one in myself after a sortie when flying in the RAF at the time.” The author adds that he wishes to keep a low profile for fear that the media “would undoubtedly link [this work] with my other known activities on SDI [strategic defence initiative, better known as the Star Wars project], BM [ballistic missile] defence etc.”

What becomes clear from the files is a fear among officials of further inciting UFO-mania. “We need to be very careful about expanding ‘UFO’ business and thereby sending the public a misleading message about the extent of the MoD’s interest,” says one memo.

Another says: “Keep the classification low – I suggest restricted – in order not to put it in the public domain but at the same time prevent the criticism that MoD is hiding something under the cloak of secrecy.”

The report was completed in 2000 and duly discounted the possibility of alien spaceships over the UK, giving DI55 the grounds it needed to no longer accept reports of sightings. But what DIS did next could only fuel the imagination of conspiracy theorists. It destroyed the files on which the report was based, and even the database in which they had collated for analysis.

David Clarke, a research fellow and lecturer in journalism at Sheffield Hallam University and a specialist in modern folklore, obtained the files from the MoD after a freedom of information battle. “They always say that the public gets the wrong idea about UFOs, but they’ve actually encouraged that themselves by destroying the files. They’ve actually encouraged conspiracy theorists through their own paranoia,” he said.

The MoD does not officially respond to queries relating to UFOs. It closed its UFO desk in 2009 after it was decided it served no defence purpose and that it took staff away from more valuable defence-related activities.