Nobody dies at a funeral. The dead arrive that way, and the living are supposed to leave that way. But the Rev Willie Maxwell walked into a funeral that he never left. It was 18 June, 1977, and Maxwell, a rural preacher living just outside Alexander City, Alabama, was at the House of Hutchinson Funeral Home – not to conduct a service, but to attend one for his 16-year-old stepdaughter, who had been murdered the week before. It was a stifling day, and with only one storey in the funeral home, there was nowhere for the heat to rise. Ceiling fans shuffled air around the chapel, and ushers offered paper fans to each of the 300 mourners as they made their way to the pews. Up in front of them, Shirley Ann Ellington’s slight body rested in an open casket.

After some hymns and a eulogy extolling the teenager’s warmth and energy, the mourners came forward to say their goodbyes, including the preacher and his wife. She was so overwhelmed after looking into the casket that she had to sit down, so her husband led her back to their pew. Despite the tragic circumstances, the couple had attracted more stares than sympathy that day: many of those in attendance believed that, far from grieving for his stepdaughter, Maxwell had been the one who had murdered her. As the last few mourners filed up, one of Ellington’s siblings pointed at him and shouted loudly enough for everyone to hear: “You killed my sister and now you gonna pay for it!”

Before anyone in the chapel could react, a man pulled a pistol from his pocket and fired three rounds at Maxwell’s head. The shots came from close range. The man with the gun was in the pew in front of his target; had he been any nearer, he would have brushed the preacher’s moustache with his Beretta. Maxwell tried raising his handkerchief to wipe away the blood, but he died before the white cotton touched his face.

The sound of the pistol sent the mourners stampeding through the doors and diving out of windows, but the man who fired it, Robert Burns, stayed inside, and when two police officers showed up to arrest him, he promptly confessed. “I had to do it,” he said on the way to the police station, “and if I had to do it over, I’d do it again.” Burns was related to Ellington, and like a lot of people around that part of Alabama, he’d grown increasingly afraid of Maxwell.

One by one, two wives, a brother, a nephew, a neighbour, and finally a stepdaughter of Maxwell had died under circumstances that nearly everyone agreed were suspicious and some deemed supernatural, yet the criminal justice system had proved powerless to stop him. Lucrative life insurance payouts on the dead, combined with rumours that Maxwell was practising voodoo on the victims and on the law enforcement officers who should have been arresting him, left people worrying about who might be his next target. So widespread was the fear that nobody who came to Burns’s trial a few months later wondered why he had shot the preacher. They only wondered why no one had done it sooner.

That trial was a sensational one, attracting standing-room-only crowds and journalists from around the country. But the most famous writer in the courtroom that autumn was one that nobody recognised – a novelist from a few counties over who, almost two decades after publishing her only book, was finally trying to write another.

Harper Lee, pictured on the porch of her parents home in Alabama, in 1961 – a month after winning the Pulitzer for To Kill a Mockingbird. Photograph: Donald Uhrbrock/Time & Life Pictures/Getty Image

Harper Lee always said that she was “intrigued with crime”. She grew up surrounded by stacks of the magazine True Detective Mysteries and the stories of Sherlock Holmes, watched trials from the balcony of the local courthouse as a young girl, studied law at the University of Alabama, and helped her childhood friend Truman Capote with the reporting and research that became his 1966 true crime classic In Cold Blood. And then there was her own masterpiece: To Kill a Mockingbird, a seemingly sweet coming-of-age story that crescendos into one of literature’s most memorable courtroom dramas.

But all of that was long in the past by the time Burns shot and killed Maxwell. At the time, Lee was living, as she had been for most of her adult life, on the Upper East Side of Manhattan, hiding in plain sight so successfully that her building’s door buzzer could read “Lee-H” without causing anyone to ring it. Her novel had won the Pulitzer prize, sold millions of copies, and made her extravagantly wealthy, but success did not suit Lee. She lived as frugally as if she were still a starving artist, was allergic to the press and publicity, chafed at the ongoing interest in her private life, and struggled to live up to the critical and popular expectations for her work. In one forlorn letter, she told a friend that “Harper Lee thrives, but at the expense of Nelle” – the name she had gone by as a child, and that those closest to her still used.

By the 70s, Lee’s friends worried about her drinking and her emotional volatility, while everyone worried about her struggles with writing. When she discussed her work, she sounded like some self-mortifying mystic, valorising suffering and solitude: “To be a serious writer requires discipline that is iron fisted,” she once said. “It’s sitting down and doing it whether you think you have it in you or not. Every day. Alone. Without interruption. Contrary to what most people think, there is no glamour in writing. In fact, it’s heartbreak most of the time.”

That heartbreak was obvious to those who knew her. Her neighbours had learned that a late-night knock on the door was likely to be Lee, made brazen by alcohol and looking for more of it; at least once, she confided to one of them that she had just impulsively thrown 300 pages down the incinerator. She told stories about other manuscripts, too, including one that was allegedly stolen from her apartment, but mostly she avoided any talk of writing. Family and friends knew to avoid the subject, too, not only with her but with the world; at most, they would say that Lee was always at work on something. For more than a decade, though, what she was writing, if she was actually writing anything at all, remained a mystery.

Lee had made her name as a novelist, but she wanted to write a true crime book as good as In Cold Blood

That was until Lee heard about Maxwell and decided to turn his story into a book. In so doing, she would not only prove something about herself – that she was not, after all, a one-hit wonder – but also about the genre of true crime. The daughter of a newspaper editor, Lee believed strongly in the difference between fact and opinion, between fiction and non-fiction; part of what she so despised about the press coverage of her own life was its many inaccuracies and distortions of the truth. (She sent more than a few annotated profiles of herself to friends, with errors circled and outrage expressed in the margins.) And while she had refrained from publicly disparaging Capote’s “nonfiction novel”, she privately thought it was an abomination. In a letter to his fact-checker at the New Yorker, she lamented: “Truman’s having long ago put fact out of business had made me despair of ‘factual’ accounts of anything.” And yet it seems she hadn’t fully despaired of factual accounts. Despite having made a name for herself as a novelist, she decided to write a true crime book, one in which just-the-facts journalism might make for a page-turner as good as In Cold Blood.

When Lee arrived in Alexander City, she settled into a cabin on the shores of Lake Martin. She got to town in time to attend the trial of the man who shot Maxwell, watching as he was acquitted by reason of insanity – represented, incredibly, by the same attorney who had previously represented the preacher in all of his civil and criminal cases. After that, she wore down the tyres on her car and the soles of her shoes travelling around three counties to interview police officers, attorneys, judges, local clergy, court reporters, relatives of the victims, and just about anyone who had ever met Maxwell. She was so consumed by the story she was reporting that, when she adopted a stray cat, she took to calling it “The Reverend” – who, in addition to his other alleged powers, was rumoured to be able to turn into a black cat.

Lee found no evidence of that ability, unsurprisingly, nor of any other supernatural powers. “He might not have believed in what he preached, he might not have believed in voodoo,” she once wrote, “but he had a profound and abiding belief in insurance.” In the course of her reporting, she turned up dozens upon dozens of insurance policies, all taken out by Maxwell, seemingly without the knowledge of the insured, with his home address as the correspondence address and naming himself as the beneficiary. The more she learned about the earlier deaths, the more convinced she became that at least five of them were murders, even though he had never been convicted of any of them.

As Lee continued her research, she swapped her lakeside digs for a room at the Horseshoe Bend Motel, not only because it was the nicest place in Alexander City, but because it was owned by her niece’s husband. She set up shop exactly as she had done all those years before with Capote at the Warren Hotel in Kansas, and the busboy who delivered her meals in exchange for 50 cent tips watched as stacks of paper piled ever higher on her desk: trial transcripts, death certificates, insurance applications and claim forms, maps, newspaper clippings, case files, jury lists.

As those stacks grew, so did Lee’s excitement about the book she was by now referring to as “The Reverend”, and she began calling friends to brag about her progress. She told Gregory Peck, the actor who had won an Oscar portraying Atticus Finch in the film adaptation of her novel, that she would have another part for him soon, and she invited an editor from Manhattan down to visit the setting of her next book. Lee made new friends, too, attending parties with reporters at the local paper, going for cocktails at the nearby country club, and holding court at the motel bar. “You simply can’t beat the people in Alex City for their warmth, kindness, and hospitality,” she wrote in a thank you note to some local hosts, before claiming that she would “be coming back until doomsday” to work on “The Reverend”.

The social aspects of reporting were good for Lee, and at first it seemed as if she had left her demons behind in New York City. But for all her initial energy and enthusiasm, her new mockingbird eventually came to seem instead like an albatross. Despite amassing more than enough Maxwelliana for a book, she could not get traction in the prose. That’s a common enough problem, of course; nothing writes itself, and no matter how many pages a reporting trip yields, the one that matters most always starts out blank. Everyone told Lee that the story she had found was destined to be a brilliant book. But no one could tell her how to write it.

Years disappeared into the struggle to do so. Lee tried to write in her apartment on East 82nd Street. She tried to write at a friend’s country house in Vermont. She tried at her sister’s house in Monroeville, and at her other sister’s house across the state in Eufaula. Everywhere she went, she brought along The Reverend, and everywhere she went, people asked about her progress, or lack thereof. In 1981, she lamented to Peck that with her first novel, “nobody cared when I was writing it”, but “now it seems that my neck is being breathed on”. She complained about how her “agent wants pure gore & autopsies” and her “publisher wants another best-seller”, but all she wanted was “a clear conscience, in that I haven’t defrauded the reader”.

Harper Lee in her father’s law office in 1961. Photograph: Donald Uhrbrock/The LIFE Images Collection/Getty Images

Ethics and expectations conspired with Lee’s pre-existing tendencies towards both perfectionism and despair. “A good eight-hour day usually gives me about one page of manuscript I won’t throw away,” she once said. She was blocked, and as she had before, she found a familiar solution: as one of the attorneys in Alexander City said, “she’s fighting a battle between the book and a bottle of scotch. And the scotch is winning.”

As more time passed, rumours multiplied. Some of Lee’s sources in Alexander City claimed that she told them she was nearly finished with The Reverend. One understood that the book was already on its way to the printers. Friends of hers in Tuscaloosa heard that she’d written the whole thing but that the publisher had rejected it because the racial politics of a black serial killer were “too sensitive”. Others said that Lee was worried about her safety because the reverend allegedly had living accomplices, or that she was worried about being sued because the story was so incendiary, so she had shelved the completed book with instructions for it to be published after her death. One person claimed to have seen a cover, another had a chapter, someone else claimed that one of Lee’s sisters had read the whole thing and declared it better than In Cold Blood. But no one could ever produce a manuscript.

For the last four years, I have retraced Harper Lee’s steps in Alexander City and everywhere else she went. I searched archives and libraries around the world, locating more than 100 new letters of hers along the way, interviewed her friends, family and neighbours, including some who had never spoken publicly about her before, and tracked down all of the people who expected to appear in The Reverend or, in many cases, their survivors. Some days, looking for information on Lee’s missing manuscript seemed like chasing a shaggy dog – for instance, when I was trying to track down the estate of the deceased University of Alabama professor who once received a letter from his brother indicating that, according to a close confidant of Lee’s, she had written the entire book. I eventually found his heir, but not a copy of the letter. I never did find the doctor Lee allegedly talked to about undetectable poisons.

I had spent years trying to reconstruct her work, and here in the briefcase were her documents and research

On another day, though, a nonagenarian former senator would call in the middle of the night to tell me about how long it had taken her to find Lee, back when the writer was holed up at the Horseshoe Bend Motel, and then regale me with details of their conversation. There was the court reporter I’d been told was dead who not only turned out to be alive but produced for me the cheque Lee had written her in exchange for a trial transcript. And then there was the day that the family of Tom Radney, the attorney who had defended both Maxwell and the man who shot him, contacted me to say that they had some news: an oversized briefcase belonging to Radney had been found among Lee’s effects and was being returned.

It was a shocking discovery, one that came after years of looking for all the papers that Radney had loaned Lee four decades before. The briefcase had been in her possession until her death; it was covered with dust, but brimming with legal files and Lee’s other materials – everything from the catalogue of an occult bookstore where she bought voodoo books to the warranty for the tape recorder she used while reporting on the Maxwell case. I had spent years trying to reconstruct her work on The Reverend, and here were her files and photocopies, documents and research. She had even saved a scrapbook of newspaper clippings from the original murders.

Perhaps most significant, though, was a single typed page of her notes. Dated “Jan 16 ’78”, it detailed Lee’s interview with a sister of the first Mrs Maxwell, who said that she knew for sure that the preacher had killed not only her sister but also his own brother. That one page confirmed something I had suspected all along about Lee’s reporting: the yellowing sheet was identical to the notes she had typed for Capote in Kansas all those years before. Those other typed pages are archived at the New York Public Library; now, for the first time, I had material evidence of their perfect echo with her reporting in Alexander City.

Like so much of what I had uncovered about Lee’s work on The Reverend, these discoveries felt precarious and precious. A year after I interviewed her, the senator who had spoken with Lee died; the court reporter died not long after that. Lee herself was 89 at her death, and many of the people who knew her best predeceased her. Others, in all likelihood, will not be with us much longer. Memories and documents are often similarly fragile. In fact, of all the pages Lee is said to have produced of The Reverend, only four have been found: a draft chapter wherein Maxwell calls his attorney in the middle of the night to ask for help when the police come to arrest him for the murder of his first wife. Lee titrates the plot like the best crime writers, but where the book went from there is anyone’s guess.

And people love to guess. Not only about how much Lee wrote, but about whether she destroyed what she had written; not only when she stopped writing, but why. After the unexpected publication of Go Set a Watchman shortly before her death, rumours swirled about the existence of other manuscripts as well, including the missing true crime book. Because her estate is sealed, her literary assets, including whatever else exists of The Reverend, remain unpublished and unknown. Perhaps there is an entire manuscript, or there are some additional chapters, or more pages of her reporting notes, possibly even the tapes of her interviews, but until the archive is unsealed, Lee’s The Reverend will continue to be the subject of as much “rumor, fantasy, dreams, conjecture, and outright lies” as Maxwell himself once was. In a strange symmetry of author and subject, the mystery of how he got away with murder became forever entwined with the mystery of what happened to Harper Lee’s book.

• Furious Hours: Murder, Fraud and the Last Trial of Harper Lee by Casey Cep is published by William Heinemann on 16 May.