In November 2001, a young German historian working at the University of Glasgow paid a flying visit to the Public Record Office, as the UK's National Archives were then known, as part of his research into the battle of the Atlantic, the maritime campaign that was waged throughout the second world war.

Sönke Neitzel had frequently travelled to the archives at Kew, south-west London, picking up a few worthwhile scraps on each visit.

"But this time it was different," recalled Neitzel. He had stumbled across a treasure trove: transcripts of conversations betwen captured German servicemen at prisoner-of-war camps in England. There were thousands of of the conversations, all covertly recorded in one of the most extraordinary secret surveillance operations of the war. "I had to be one of the first people [since the war] to have ever held them in his hands," he said.

Declassified five years earlier, the transcripts had been filed away at Kew and appear to have been largely unread. This is not particularly unusual: an enormous number of documents are transferred to the national archives each year. Many remain, in effect, undiscovered: hidden from history in plain sight.

Neitzel's find resulted in two remarkable books (both conventionally published), Tapping Hitler's Generals, containing transcripts edited by Neitzel, and Soldaten: On Fighting, Killing and Dying, which he wrote with the social psychologist Harald Welzer.

Meanwhile, the British historian Helen Fry had been approaching the same subject from the other side of the eavesdropping devices: by studying the activities of the Germans and Austrians serving in the British army who listened in on the conversations.

Fritz Lustig, a second world war 'secret listener'

One in seven of the 75,000 Germans and Austrian refugees who came to Britain between 1933 and 1939 enlisted in her armed forces after war was declared. Many were Jews, others were political enemies of the Nazis; some were artists who fled after being denounced as "degenerates". Fry has chronicled the experiences of these men and women in several books.

In Churchill's German Army, published in 2007, she wrote about those German and Austrian men and women who fought against their homelands in each branch of the British services – "giving something back to Britain for saving my life" – as one put it.

Her latest, The M Room – which she has self-published on Amazon – tells the stories of refugees recruited to the Combined Services Detailed Interrogation Centre (CSDIC), that monitored the PoWs' conversations using highly sensitive microphones developed by Post Office engineers based at Dollis Hill, north-west London.

Fry explains: "A few years ago I promised WW2 veteran Fritz Lustig that I would tell the story of the top-secret unit that he was drafted into during the war. What I uncovered about the work that he and other 'secret listeners' did in bugging the conversations of German prisoners-of-war astounded me. I had stumbled upon an incredible untold history that was as important for winning the war as Bletchley Park. And yet no publisher believed in it enough to take it on – something which continues to surprise and baffle me.

"With Fritz approaching his 94th birthday I owed it to him to get the story out in his lifetime, and I decided to self-publish."

CSDIC mounted electronic surveillance operations codenamed M Cover – M was for mic – at three prisons housed in large country houses in the south-east of England. The secret listeners eavesdropped on their countrymen from "M rooms", hunched over monitors and recording key moments of conversation on 78rpm discs.

Listening in day after day, they discovered many of their targets' deepest fears and secrets. But they never set eyes on the PoWs.

Fry introduces us to a handful of the scores of secret listeners: Lustig, a cellist from Berlin; Rudi Oppenheimer, a lawyer from Nuremberg; Peter Ganz, a boy from Mainz who had survived a spell at Buchenwald; and George Pulay, a 20-year-old from Vienna: all serving in the British army, all determined to play their part in the defeat of Nazism, and all doing so by eavesdropping on their compatriots.

"We sat at tables fitted with record-cutting equipment," Lustig recalls. "We had a kind of old-fashioned telephone switchboard facing us, where we put plugs into numbered sockets to listen to the PoWs through our headphones. Each operator usually had to monitor two or three cells, switching from one to the other to see whether something interesting was being discussed. As soon as we heard something we thought might be valuable, we pushed a lever to start a turntable revolving, and pulled a small lever to lower the recording head on to the record."

Inmates were often held in two-man cells at CSDIC's prisons, so that the listeners could be sure who was talking to whom. The listeners were expected to possess a mastery not only of the technology employed by each of the German armed forces, but of the country's different dialects. Stool pigeons were occasionally thrown into the mix, to steer to prisoners' conversations into more interesting areas.

German generals – prisoners of war at Trent Park

In this way, Fry discloses, CSDIC's listeners discovered that SS concentration camp guards had mutinied several years before the war, and recorded conversations about human stud farms, at which women were encouraged to breed with SS officers.

More importantly, in May 1943 they heard two German officers discussing the huge cost of German experiments with rockets and jet engines. That idle chatter helped the RAF to make sense of a series of bewildering aerial surveillance photographs, and bombing raids were promptly launched against the V2 rocket plant at Peenemünde on Germany's Baltic coast.

More than 10,000 German and some 560 Italian prisoners passed through CSDIC's three surveillance prisons. Most of the prisoners appear to have known that their cells may have been bugged. Certainly, they were briefed about the risk of being secretly monitored as part of their training on how they should conduct themselves if captured. But after a little while, many dropped their guard and began talking freely. CSDIC prepared almost 17,000 transcripts of German conversations, and 1,940 from the Italians.

CSDIC's Jewish German recruits became some of the first people to hear detailed descriptions of the slaughter as the Wehrmacht rolled eastwards across Poland and the Baltic states: accounts of entire communities being gunned down beside ditches or gassed inside trucks.

One officer was recorded explaining that he had angrily complained about a mass shooting in Latvia – adults shot while lined up beside a trench, children held up by their hair and shot in the head – because the corpses were contaminating water supplies. "I … went to this Security Service man and said: 'If you shoot people in the wood or somewhere where no one can see, that's your own affair. But I absolutely forbid another day's shooting there. We draw our drinking water from deep springs; we're getting nothing but corpse water.'"

Another describes peeking inside the back of one of the Nazis' mobile gas chambers or Gaswagen: "Only a glance. They must stand up as I didn't see any seats. Once they were inside they were shut in, the engine was turned on, and in a few minutes they were dead."

The M Room's cover shot – a listener at Trent Park

Nor were all the prisoners mere bystanders. One general, Dietrich von Choltitz, was recorded admitting that he carried out orders to murder Jews: "We all share the guilt. I misled my soldiers into believing this rubbish. I feel utterly ashamed of myself."

As they recounted their war stories in their matter-of-fact way, both prisoners and the secret listeners became immersed, every day, in extreme acts of cruelty and violence. The listeners knew that they could be eavesdropping on accounts of the deaths of their own families and friends. After the war, both Rudi Oppenheimer and Peter Ganz learned that close relatives had been murdered. "We all tried to be professional in our approach," says Lustig. "There was the expectation that justice would eventually be done."

But justice was not always done: many of CSDIC's prisoners escaped war crimes trials. Lustig and his comrades were informed, falsely, that their recording could not be used as evidence as they breached the Geneva conventions.

The truth, as uncovered by Fry, is immediately recognisable to anyone following the debate about the use of telephone intercept evidence in English courts today: the War Office decreed that the recordings could be used to guide war crimes investigations but could not be adduced in evidence, as to do so would give the game away the next time the country went to war. "The basic security, from the point of view of the future, of CSDIC methods as developed in this war must be considered," concluded a subcommittee of the Joint Intelligence committee. "The greater effect of playing back these recordings in court, the more deeply will the future use of CSDIC's be compromised."

Sneakily, but perhaps unsurprisingly, the British authorities also decided that barristers who were to cross-examine war criminals should be allowed to listen to the recordings, in confidence, but that the defence would learn nothing of their existence.

One omission from The M Room is that it makes no mention of the brutalities that were sometimes employed at CSDIC prisons: interrogators at its Cairo outpost are now known to have boasted of the cruelties they inflicted in order to "turn" captured agents, while some inmates at its postwar prisons in Germany were slowly starved to death and others tortured with implements recovered from Gestapo jails.

And as it is self-published on Amazon, The M Room lacks both the polish that a professional editor would have brought, and an index.

But this barely detracts from a book that makes a invaluable contribution to the history of the period, and to Fry's own unique body of work, chronicling the efforts of wartime Britain's German warriors.

For Fry herself, self-publishing has been far from second best: "This route meant that as soon as the book was complete it was up for sale within a matter of days. In the six months since its publication it has enjoyed unprecedented media coverage, following a live interview on BBC's One Show in January. It became the subject of a docu-drama for Channel 4, Spying on Hitler's Army, and subsequently claimed the No 1 spot on ITV's recent series Britain's Secret Homes.

"In a climate where publishers seem to be failing many mainstream authors, what can authors do other than turn to new options being provided by the fast-changing market? What I have learned from this self-publishing experience is that whatever the medium of publication, a darn good story always sells."

Ian Cobain's Cruel Britannia: A Secret History of Torture is published by Portobello.