Brouwenstijn's now crumbling wallet is one of hundreds stored in brown envelopes marked "effects" at the Red Cross International Tracing Service (ITS) in Bad Arolsen, central Germany. Other envelopes contain everyday objects such as a bronze powder press, whose pale pink contents briefly cloud the air when opened, a tiny lipstick embossed with the words "kiss proof", and a white rosary.

These trinkets and bits of paper are, in thousands of cases, the only remnants of lives that ended, as Brouwenstijn's did, at the hands of the Nazis, the envelopes protecting their memories.

Before the end of this year, the fragmented stories of 17.5 million individuals held in concentration and slave labour camps during the second world war are due to be revealed to historians and academics for the first time.

For decades the ITS - founded in 1943 to search for "displaced persons" - has restricted access to the largest collection of Nazi death camp records to survivors. Even so, many of their requests for information have taken years to be answered.

But last year the 11-country committee that controls the ITS voted to open up the records. Now the committee has set procedures in motion to open the files before the end of this year, once a ratification process has been completed.

In preparation, all the archive's 50m documents are in the process of being digitised and indexed for electronic use by the ITS's 330 workers, in what will amount to 7,000 gigabytes of data.

This week the Guardian was given access to the vast storehouse, with contents stretching across 16 miles of shelving.

The archivists say it is unlikely that the Bad Arolsen collection will throw any new light on the Holocaust. But studying even a fraction of the mass of information, recorded meticulously by Nazi bureaucrats - and sometimes prisoners - with the aid of typewriters, pencils or ink, on everything from the mundane to the extraordinary, allows a vivid picture to emerge of a vast number of lost lives.

Punishments

In one file randomly plucked from a shelf, there is a report by a doctor from Ljubljana held at Dachau describing a regular punishment inflicted on prisoners by their SS guards. "They tied their hands behind their backs and suspended them from the branches of the trees by their wrists so that their feet were off the ground," he wrote. "This procedure would usually take place around 11am so that every prisoner had a chance to see the men hanging when they went to the mess."

A file on Melk, a sub-camp of Mauthausen in Austria, contains a sketch by a Swiss dentist showing where he buried the personal possessions of deceased prisoners for safekeeping.

Among the more famous of the documents is "Schindler's List", which details the identities of the more than 1,000 men and women saved by industrialist Oskar Schindler, including glaziers, doctors, engineers, and seamstresses.

"Annelies Maria Sara Frank" - how the Nazis listed the young diarist Anne Frank - leaps off the page of the alphabetical list No 40 of deportations to Westerbork. Her final destination is left blank - as is known, she perished in Bergen-Belsen.

It is an archive with many tales to tell. One of the most shocking is a neat handwritten report within the pages of a frayed ledger from Mauthausen which describes how on the orders of the Reich security department in Berlin, 300 people were shot in the neck between 11.20am and 12.54pm on April 20 1942, as a present for Hitler on his 53rd birthday. All 300 names are listed.

Records for the camp Gross-Rosen in Poland, about which few documents exist, detail head lice controls, including prisoner numbers and the precise number of lice they had. These records have provided some survivors with their only piece of written evidence that they were in a camp at all, evidence which has proved vital in several compensation claims.

Records for Buchenwald give a glimpse into the bizarre day-to-day workings of the camps. A so-called "manpower report" reveals that seven hairdressers and 252 tailors were at work in the camp on January 8 1940.

There are also detailed reports of the Death Marches - the Nazi attempts to move prisoners from camps to the German heartland at the end of the war - pieced together from eyewitness accounts, detailing how many perished along the way.

The German government has been accused by critics, including Holocaust survivor groups, of refusing to open up Bad Arolsen due to fears that any move to make the information more widely available outside Germany would result in it facing further compensation claims, adding to the £42bn it has already paid to victims of the Nazi regime.

Paul Shapiro, director of the Centre for Advanced Holocaust Studies at the Holocaust Memorial Museum in Washington, went so far as to say that "preventing researchers from having access to the Holocaust documents is in itself a form of Holocaust denial".

Last generation

Others say it is urgent that the archive is opened because the last generation of Holocaust survivors are now in their twilight years, and not least because the Holocaust has become so institutionalised. But the ITS, which last year received 200,000 requests for information, insists that the reluctance is due to concern about data protection, and that it is only abiding by German law. This German propensity to do things by the book is also used to explain the fact that while it is described as a confidential service to trace missing people, rather than an archive, Bad Arolsen's information remains out of bounds.

There are also concerns about protecting individuals. "There are cases in our files of priests accused of paedophilia, and whether true or not you would not want a family stumbling across such information," said ITS's spokeswoman, Maria Raabe.

Some files detail cases of hereditary and sexually transmitted diseases, which might be considered sensitive information. "And there are cases where people just don't want to be found - who deliberately disappeared after the war, or who don't want to be traced by an illegitimate child they gave birth to or fathered," she said. "We have a duty to protect their identities."