In the fetid, druggy summer of 1969, Charles Manson, a hippy messiah who claimed to be the spawn of Christ, Satan and Hitler combined, dispatched his besotted apostles on killing sprees throughout Los Angeles.

For starters, they obscenely butchered Sharon Tate, then pregnant with Roman Polanski’s child, and a smattering of her friends; the next night, the gang slaughtered a suburban grocer and his wife, leaving a devilish-looking pitchfork stuck in the man’s chest.

Other corpses were disposed of in Death Valley or on the movie ranch where Quentin Tarantino sets his new film Once Upon a Time… in Hollywood, which ends with the Manson rampage. The killers, also cosily known as the Family, apparently chose their victims at random and didn’t bother to rob the properties they invaded. These were motiveless murders, a ritualised manifestation of evil. Or so we thought, until Tom O’Neill began asking inconvenient questions.

O’Neill’s investigations started 20 years ago with a commission from a movie magazine and they have continued ever since. Although he never managed to write the magazine article, his research monomaniacally took over his life. Deadlines were indefinitely extended and publishers eventually demanded their money back or lost their nerve when confronted by his scandalous findings; this book is only an interim report on a sticky network of lies by the LAPD, the FBI and the CIA, which joined forces in an effort to muddy the truth about Manson’s crimes.

Among Manson’s bibles was a novel about a saviour who grows up on Mars and descends to earth to supervise the apocalypse

O’Neill first accuses the prosecutor Vince Bugliosi of tampering with witnesses; Bugliosi, who turned his courtroom assault on Manson into a lucrative book, retaliates by threatening to sue the virtually penniless O’Neill for a hundred million dollars and smearing him as a gay paedophile. The next teasing trail of evidence persuades O’Neill that FBI agents colluded with the district attorney, presumably because J Edgar Hoover’s bureau saw Manson and his gaggle of sex slaves as countercultural brutes who deserved to be exterminated.

The CIA becomes involved when O’Neill starts to wonder why Manson’s parole officers were so lax in the years before 1969, allowing him to decamp to San Francisco. It turns out that while living in the psychedelic gulch of Haight-Ashbury, Manson may have been used as a guinea pig or lab rat by a bent doctor who recommended LSD to the government as a means of mind control that could penetrate the heads of informants or prepare so-called Manchurian candidates for service abroad as spies or assassins. Back in Los Angeles, did Manson employ the same methods to programme his murderous surrogates?

As it develops, O’Neill’s tale embroils an increasingly stellar cast of accomplices or enablers. There’s even a walk-on by Doris Day, the embodiment of the sanitised America ravaged by Manson. Day’s son, Terry Melcher, used her beach house to host orgies for a club of priapic buddies known as “the Golden Penetrators”; Manson, a wannabe rock star, supplied groupies to the revellers in exchange for a promised recording contract with a label Melcher managed for his mother.

No unified field theory of malfeasance would be complete without a link to the JFK assassination and the ensuing cover-up. O’Neill supplies one and to me it seems only too plausible. Inevitably, there are contemporary resonances. Manson emerged from the “divided, disorderly America” of Lyndon Johnson, a neurotic president who, like the current incumbent, declared war on his own insurgent society and blamed foreign infiltrators for mobilising resistance to him. What shocks O’Neill is “the many ways our government has deceived us”: in the half-century since the Manson killings, we have become inured, on both sides of the Atlantic, to such official mendacity.

Among Manson’s homespun bibles was a sci-fi novel about a saviour who grows up on Mars and descends to Earth to supervise the apocalypse. He and his gang behaved like aliens, but they were native stock: they confirmed “the nation’s fears about itself” in a deranged parody of the American creed – its family values, its cult of celebrity, its mad religiosity. At the end of his book, O’Neill says that, by the time he died in prison in 2017, Manson had become “a true-crime icon”. It’s not clear how ironic he intends that sacrilegious phrase to be: icons, of course, are devotional portraits of holy figures.

During his long slog, O’Neill encounters a series of bogeys who would be at home in one of Raymond Chandler’s novels. They include Manson’s defence lawyer, now a shambling, toothless derelict with a briefcase held together by string; a prison inmate known as Pin Cushion because of his uncountable stab wounds; and a CIA scientist who killed an elephant called Tusko by injecting it with 1,400 times the dosage of acid that produces “marked mental disturbance” in humans.

O’Neill’s dottiest character is himself, a lonely obsessive hoarding Manson ephemera in his cramped apartment, with 190 binders full of notes on his shelves, plus six extra stacks of unfiled documents, each of them four-feet high, littering the floor. He begins as a dutiful gumshoe like Chandler’s Philip Marlowe plodding down the mean streets, but he ends closer to the justifiably paranoid taxi driver played by Mel Gibson in Richard Donner’s thriller Conspiracy Theory. As he admits, the loose ends are still not tied up and with so many of the culprits dead they probably never will be. O’Neill’s intricately sinister “secret history” often sounds incredible; that doesn’t mean that it’s not all true.

• Chaos: Charles Manson, the CIA and the Secret History of the Sixties by Tom O’Neill with Dan Piepenbring is published by William Heinemann (£20). To order a copy go to guardianbookshop.com or call 0330 333 6846. Free UK p&p over £15, online orders only. Phone orders min p&p of £1.99