Doctor Who is the most documented programme in the history of television. It has generated hundreds of scholarly books and articles. Over 34 years Doctor Who Magazine has examined every episode, spin-off novel, audio drama and comic strip in microscopic detail. Remnants of rejected scripts from the bottom drawers of dead screenwriters have been reconstructed and recorded. The memories of production team members have been sifted by convention delegates and the makers of DVD extras. Every dispute, tantrum, writ and nervous breakdown; every all-nighter at the keyboard or in the Colony Room has been logged, archived, discussed. We – and when I say "we", I probably don't mean "you" – know that Ridley Scott was originally on the rota to design the Daleks, that Tom Baker looks weird in "The Ribos Operation" because a dog bit his face down the pub, and that the galactic co-ordinates of the windswept planet of Kastria are the phone number of the Doctor Who production office, circa 1976. If the discourse of Doctor Who were the subject of a Doctor Who story, the cliffhanger would reveal that it had evolved into a pulsing entity bent on cataloguing the universe to destruction.

Deep down, most Doctor Who fans prefer this discourse to be about provisional story titles and the limited lift capacity at Lime Grove studios. Their interviewees, however, have begun to talk about more personal matters. The honesty of the former colleagues of the first Doctor, William Hartnell, has ensured that his racism is now part of the accepted narrative of his life. His onscreen companion Anneke Wills has described how she escaped from an abusive marriage to Michael Gough into the ashram of the Bhagwan Shree Rajneesh. In the current issue of Doctor Who Magazine, Frazer Hines – who played Jamie, the only companion to merit a mention in Joe Orton's diaries – talks about the stops second Doctor Patrick Troughton would make on the drive home from TV Centre: "We'd go to three different houses on the way," he recalls. "He'd knock on the door, give this woman some money and then we'd drive off. I'd look the other way." Slowly, all those details about scene-shifters' strikes and monsters built from fox skulls and condoms are being augmented by stories of the everyday emotional sturm und drang of the people who walked through those sets and ran away from those monsters. This is not happening to the cast and crew of Casualty, because a world without Casualty would be only marginally different from this one – whereas for many of us, a universe without the Doctor scarcely bears thinking about.

JN-T: The Life and Scandalous Times of John Nathan-Turner will test the limits of that appetite for information. It is the frankest book ever written about Doctor Who, and contains material that could not have been published in the lifetime of its subject, a bookie's son from Birmingham who became the programme's longest-serving producer.

Nathan-Turner oversaw Doctor Who throughout the 1980s – its most eclectic decade, in which the style was sometimes Play for Today, sometimes Play Away. He produced a story that comprises a shot-for-shot homage to Cocteau's La Belle et la Bête, and another in which a leather-clad Beryl Reid fights Cybermen on a spaceship. He produced episodes about a police state in which the chief torturer is a robot made of Liquorice Allsorts, and others about a giggling slug who wants the galactic broadcast rights to execution videos from a planet whose rulers are fond of phrases such as: "I want to hear you scream until I'm deaf with pleasure."

This was also the decade in which the BBC's institutional indifference towards Doctor Who – a factor since its birth in 1963 – hardened into hostility, with cruel consequences for Nathan-Turner. In November 1983 the series was celebrating its 20th anniversary with a Radio Times cover and a film-length special called "The Five Doctors". Fifteen months later, Michael Grade, controller of BBC1, was publicly dismissing Nathan-Turner's production team as complacent and their work as tired, violent and unimaginative. For Grade and his colleagues, Doctor Who and its producer had become an interlocked pair of problems. "I wanted him to fuck off and solve it – or die, really," says Jonathan Powell, the BBC's former head of drama, in one of the many brutal remarks collected in Richard Marson's book. "But it had probably gone beyond solving. The only way of resuscitating it would have been to put a new producer on it – but we didn't want to resuscitate it." Had Powell and Grade known about some of the incidents described in JN-T, they might have been able to kill both producer and programme at a stroke.

Halfway through his story, Marson drops his bombshell. At the age of 17, he was dispatched to Television Centre to write a set report on a story called "Resurrection of the Daleks". After the recording, he was propositioned by Nathan-Turner in the bar. The following year, on the promise of some stills from an imminent story, Marson made an after-hours visit to the Doctor Who office, where he endured a sexual assault at the hands of Nathan-Turner's partner, Gary Downie, who worked as the show's production manager (he died in 2006). Given the age of gay consent in 1985, this constituted a double offence. Marson's account, though, sounds a surprising note of black humour: he hid from Downie in an adjoining room, readying to defend himself with the nearest object to hand – the script for episode two of "Timelash". Marson knows that for Doctor Who fans, this amplifies the indignity – episode two of "Timelash" is awful.

The evidence of his interviewees would suggest that this was not an isolated incident – that for Nathan-Turner and Downie, making passes at Doctor Who fans became something close to a social reflex. ("Doable barkers" was their gruesome term for those who aroused their interest.) The most bizarre narrative involves a wealthy fan who, in exchange for visits to the studio and the occasional souvenir from the set, kept Nathan-Turner supplied with escorts. It's both an unedifying scenario and a telling measure of the occult power of the programme: Doctor Who's producer receives a strip-search from a fake policeman; the fan takes delivery of a script used by Tom Baker. I suspect only The League of Gentlemen could do the story justice.

In a former life Marson was the editor of Blue Peter, and by his own admission, his subject's life forms a partial mirror of his own. Both built their careers in public service broadcasting; both took charge of warhorse shows for which the nation had greater enthusiasm than the BBC's management. Both received humiliating dressings-down from their employers: Marson because of trust-breaking incidents involving a phone-in, a blue-eyed ragdoll kitten and a fake competition winner; Nathan-Turner because he stayed stubbornly at the helm of a show through which his bosses were itching to send a torpedo.

Marson's book is the product of admirably dogged research. Its author has tracked down his subject's schoolmates, colleagues and the unexpected figure of a former girlfriend. He has squirrelled some rich material from unpublished correspondence and forgotten cassette tapes. But his book is, in essence, a distended fanzine article, and not everything ghastly on its pages can be ascribed to his subject. He has, for instance, carefully transcribed his interviewees' bickerings about how often Nathan-Turner attended the STD clinic – and who had the honour of driving him there. It's hard to imagine a more experienced biographer making the same decision – or sharing his turn of phrase. I doubt such a biographer would use the expression "shit-storm", or would describe how a subject made someone "his 'bitch', literally as well as figuratively". The observation is as meaningless as it is vulgar.

Middle-aged fans at the convention bar might be tempted to process this material into black humour. (A fashion company is already marketing the "Doable Barker" T-shirt, the phrase spelled out in the correct neon font of the Doctor Who titles commissioned by JN‑T.) The press, the public and the BBC, however, may respond very differently. Though he couldn't have known it when he began to write his book, Marson has now added the name of John Nathan-Turner to the conversation that began when ITV screened its documentary about the crimes of Jimmy Savile. Whether the man deserves that place should be decided by those who have genuine grievances against him. "Although I did meet some people who felt that their treatment at the hands of John and Gary was inappropriate," Marson concludes, "it would not be true to say that I've found anyone willing to testify to coercion or abuse." The abuse of power, however, is painfully clear – and this is the real subject of this book.

Doctor Who is often described as a cult. It is certainly much more than a TV programme. It has even worked its way into the consciousnesses of people who don't like it very much. For those who do, it can form the unifying principle of their lives – which confers a priestly power on some of those whose names have appeared on its credit roll. Several actors who have played the lead, for instance, report that fans sometimes unburden extraordinary intimacies as they present their autograph book for signature at a convention. The Doctor, they're often told, is the father figure who will never let you down. He is never cruel, never self-serving, never unjust. He is the man who will rescue you from the tyrants, the bullies and the exploiters – even if, in the real world, they prove impossible to avoid.

• Matthew Sweet's The West End Front is published by Faber.