In archives dotted across the cities of the former East Germany, there sit mile after mile of shelves stuffed with paper documents, photographs, film reels and audio recordings. Buried in this mass of material are records that have damaged or even ended the careers of high-profile politicians, authors and other public figures in Germany – and could do so again in the future.

These are the archives of the Stasi, the German Democratic Republic’s infamous secret police, known officially as the Ministry of State Security. To put the size of this unique record of everyday collaboration and resistance in context, the entire German state since unification in 1870 has produced 300km (205 miles) of documents. The Stasi managed, over the course of just 40 years and operating in only about a third of German territory, to create at least 111km of material. “The Stasi were like a vacuum cleaner”, hoovering up information on every aspect of life in the GDR, says Karsten Jedlitschka, a research manager at the archives.

Less well known than the archives’ vast scale is the huge ongoing operation to regulate what information is released into the public domain. Nearly 500 officials, working across 13 locations, are employed to assess requests for files. They find, read and redact documents depending on how sensitive they are.

Even though Stasi informants are dying off as time passes, news of past collaboration with the secret police is still a reasonably regular feature in German public life. After all, the scale of surveillance was huge: one report estimated that one in every 100 people in communist Germany was an informant. Berlin’s housing secretary quit in January after coming under pressure for not declaring in a previous job that he had once been a Stasi cadet. And German newspapers spent the run-up to the recent federal election digging into the Stasi past of candidates for the nationalist party, Alternative for Germany – which, nevertheless, went on to win 13 per cent of the vote.

But one cannot walk into the archives and simply ask if a public figure was an informant. Access is governed by a complex set of rules overseen by a federal commissioner, as set out in a 1991 act that East Germans insisted on post-reunification. They reflect a tricky balance between a desire to achieve justice and clarity about the past and the need to protect personal information.

Researchers and journalists therefore have to propose a broader topic of research, such as the operation of the Stasi in a particular village church. The more detailed background research the applicant has done, the more likely the proposal is to be approved, according to Jedlitschka.

If the released files reveal that, for example, a prominent bishop was once an informant, then they can be publicly unmasked. But what researchers cannot do is ask the archive for any and all files relating to that bishop, as this would be classed as a fishing exercise. Information about victims of the Stasi is also redacted, although scholars based at universities and other research institutions (but not journalists) are allowed to see victims’ names in the original files as long as they do not disclose them publicly.

These rules lead to a kind of cat-and-mouse game between journalists and the archive. Jedlitschka says that reporters are forever trying to formulate topics that unearth “personal information” on current public figures. “Personal information is accessible under certain conditions, but generally we protect privacy,” he says – although a “creative journalist” could probably come up with a historical topic that yields files pertaining to modern-day politicians. “We notice that as soon as somebody gets into a [political] position, research requests start to come in about this particular person,” says Susan Lüdecke, who works in the department that handles requests from historians, journalists and others.

Reporters have come after Angela Merkel’s files many times, the archivists say, with no results, as the agency has not been able to give out any records. On a talk show in 2009, Merkel revealed that the Stasi had tried unsuccessfully to recruit her during her youth when she applied for a job as a physicist. Information about this episode “is not available to the public” because the “protection of privacy” takes precedence here, says Jedlitschka. Nevertheless, “if there were any indications [that she was an informant] they would have come out by now”, Lüdecke says.

Sometimes journalists try to “dig in the mud” around politicians’ parents, but such requests are also rejected, Jedlitschka says.

Even for informants whose identities are revealed, there is some protection. “Personal, private sphere” information, such as sexual orientation, would be redacted, explains Lüdecke. “There are a lot of stories of informants…who were blackmailed into doing it,” she says. “Even informants have a right to an intimate sphere.”

But it is clear that the rules protecting access to the archives mean that some spies have gone unmasked. Douglas Selvage is an American historian who works as an in-house researcher for the archive’s Education and Research Division (he keeps a hefty safe the size of a fridge-freezer in the corner of his office for storing classified documents, although he stresses that this is very rarely necessary for Stasi documents – and the safe is currently empty). While researching a book about a KGB disinformation campaign to spread the idea that the Pentagon created the Aids virus, he came across two spies who had worked in West Germany for the Stasi’s foreign intelligence division, the HVA.

One of those is currently “active in local politics in Germany”, and the other “provided hundreds of pieces of information from the Western pharmaceutical industry, including things…that could also be used for biological weapons”, and this intelligence was often sent on to Moscow. As their cases were not sufficiently closely related to the topic for which he had been granted access to the records he could not publish the information. However, a new book on a topic relating to their activities might be able to name them, he says.

For archivists, it is “not our job” to hunt down informants, says Jedlitschka. In the early days of the archives, journalists might have gotten a “hint” from an archivist who uncovered some bombshell revelation, or the tip might have been passed on to an internal researcher, who could then dig deeper. Jedlitschka says that the days of such tip-offs are over, but Times Higher Education was also told that the practice still goes on.

Some are frustrated with the archive’s approach. Hubertus Knabe is the director of the Berlin-Hohenschönhausen Memorial, a former Stasi prison where opponents of the communist regime were isolated and interrogated. His office, which overlooks the compound’s drab grounds, surrounded by barbed wire, still has two doors, one opening in and the other out. The space in between would originally have been filled with sound-proofing material, he says, so that conversations could not be overheard.

“We spend…more than €1 billion on this office [the Stasi records agency], but we still don’t know who was collaborating with the Stasi. This is a little bit unsatisfying,” he says. For him, there should be a register of everyone who collaborated.

Knabe’s museum is trying to build up a picture of the prison’s inmates, but the Stasi archives’ redaction of victims’ names in files has made this very difficult. “You normally get copies of the files that you can’t understand,” he complains. When the museum was opened in 1994, it asked for a list of former prisoners, but files were returned with all names redacted. “It was useless for us,” he says.

In 2006, there was a major change to the Stasi Records Act, allowing historians working for public institutions to access the names of victims even without their consent. This has made the situation “much better”, Knabe concedes, although researchers still are not allowed to remove records with unredacted victims’ names on them, obliging the researchers to copy them out by hand at the archives.

Another irritant for historians is that the law grants access to files but not to the registers that catalogue them, because these contain personal information about victims. “It’s like a lottery. You ask for some information, you get some papers; but you don’t know, and you can’t really check by yourself, how it was found, if this is really everything, if they looked in every register,” Knabe says.

And the sheer amount of work involved in sifting through the records means that the wait for files can be painfully long. For example, Knabe wants information about inmates at the prison hospital: a tricky undertaking because medical data is so sensitive. “We think that a lot of these illnesses were produced here at the prison, so it’s useful for getting to know the methods of the Stasi,” he says. To process such a request, archive officials must check who is mentioned in the files, redact names as necessary, and then make new copies (this is necessary because if you black out a name, you can still read what is underneath it by holding it up to the light, he explains). This whole process can take six months for one prisoner; there were about 3,000 inmates in the hospital.

Still, the archives have become “more and more liberal” about divulging information, Knabe notes; for instance, he was recently granted access to bugging records of a Stasi doctor at the prison hospital, which he had previously been denied.

This is perhaps because, as time passes, the sensitivity surrounding the Stasi records is waning. “Stasi informant unmasked!” headlines do still make the newspapers, but not nearly as often as they once did. And some of this “gotcha journalism” has come to be seen as unfair, says Selvage. One of the most famous revelations was that, from 1959, the East German novelist Christa Wolf had reported to the Stasi on colleagues. However, by 1962, the Stasi had broken off contact, and, from 1968 to 1989, Wolf was “someone who was spied upon, not somebody who was spying”, Selvage says.

“There are [also] people who signed and said they would be an unofficial collaborator, and then they never reported on anybody,” he adds. “There are other people who just formally went to sessions, and just said positive things about people.”

The other problem with naming and shaming informants is that the East German Communist Party’s system of repression extended well beyond the use of secret police. The Stasi, Selvage argues, “got involved when they wanted to make a point, but not in every case of dissenting behaviour”. If there were suspicions about someone’s loyalty to the system, the party could block them or their family from gaining university posts or certain professional positions, he explains.

“In some ways, the party, after [the fall of the Berlin Wall in] 1989, used the Stasi as a scapegoat,” Selvage says, with former party officials blaming them for the oppression and spying conducted by the system as a whole. And the German public, he adds, bought into that narrative.

The Stasi, he says, “are one group of those responsible for repression, whose records have been opened…The other group responsible, the party, their records regarding individual cases of repression can remain closed or unavailable because they contain personal data [or because] there is insufficient funding locally to process the files, or the files have been destroyed.”

Be that as it may, researchers’ frustrations with accessing Stasi material can be expected to continue to recede as the communist period recedes further into history and as the fallout from revelations affects fewer and fewer living people. There is now discussion in the German parliament about whether the Stasi records should become an official part of the national archives, perhaps from 2019 or 2021. The restrictions on accessing records would still be likely to apply for another decade or two, but it would be an important symbolic move.

Knabe puts it in terms of what, given the threat of nuclear war that overhung the whole communist period, is an apt comparison between Stasi material and radioactive residue. “After 50 years,” he notes, “it gets less dangerous.”

For the record: handling sensitive information around the world

Germany is not the only country that is sitting on a massive cache of potentially incriminating records from an oppressive former regime. The rest of Eastern Europe has also had to work out how to treat its communist archives, and, on the whole, has made them much more freely accessible than in Germany, with all the implications that has for privacy and public access.

A similarly open approach has been taken in Latin America. Kate Doyle is a senior analyst at the National Security Archive, a US-based organisation that campaigns for government openness. She has worked on projects to open up archives in Paraguay and Guatemala that document the detention and torture of political opponents by the army and the police during periods of civil wars in the 1970s and 1980s.

But the accessibility of the archives is “not an expression of the states’ will to address the past”, she says. In both countries, the records were “forced open” by civil society. In Paraguay, a “huge collection” of records about police action against dissidents was discovered in a police station after an anonymous tip-off, while in Guatemala, similar records were discovered accidentally in a police facility.

Nevertheless, “there was a deliberate decision in both these countries to provide total, unrestricted access” to the records: even those that include “lies” made up by the regime to discredit opponents, Doyle explains. This is mainly because to confront the violence that took place, “these records and the history of that period have to be completely transparent”. But it also relates to the fact that the two countries “don’t have the capacity they have at the Stasi archive” to police access.

Doyle has not yet heard of any public objection in Guatemala to the free availability of the archival information. However, surveillance in the country by the police tended to be less about people’s private lives and more about their political activities. That many leftist dissidents “came out” as former members of armed guerrilla groups after Guatemala’s peace accords in 1996 also took the sting out of the opening of the records, which often name members of such groups, Doyle adds.

The Guatemalan archives have been used as evidence to prosecute a small number of people – five or six, according to Doyle – for human rights violations, and the archives now have investigators specifically to comb the files for evidence of criminal conduct. They can provide a “window into…the mechanisms of state violence and state repression,” Doyle says.

But while other countries may take different approaches, the Stasi archives, with their deeply considered rules and complex machinery, remain the first port of call for other countries mulling how to open up their painful histories. After the Arab Spring, for instance, a Tunisian delegation visited as part of its deliberations about how to make accessible the records of the overthrown regime’s secret police. And, in September, the archives hosted a delegation from Albania, a country seeking to come to terms with its own communist past.

David Matthews