We tell many stories about James Yancey.

There’s the one about how in the mid 1990s, as Jay Dee, he perfected sampling and helped reinvent the sound of A Tribe Called Quest. Or the tale of Questlove taking inspiration from his drumming style for the grooves of D’Angelo’s Voodoo. Or the one about making the 2003 low-key underground classic Champion Sound with fellow maverick Madlib. But best known of all is the story of how he crafted the definitive beat tape statement, Donuts, while on his deathbed. It’s through these stories, and countless others, that we elevated Yancey after his death to the top of hip-hop’s production pantheon. Ignored in life, his oeuvre was lauded in death; a lifetime of innovations reduced to a catchy tag line on a T-shirt. In our rush to set him stone, we forgot that Yancey had spent his life advocating forward movement, not stagnation. The intent was lost and man became myth.

There is another story we tell about Yancey, about a lost solo album for major label MCA, part of a deal signed in the early 2000s at the peak of his brush with the mainstream. Yancey’s MCA album would vindicate his until-then shadowy presence and confirm him as the producer’s producer par excellence. But it wouldn’t do so in an obvious way, because for this album Yancey had opted to put pen to paper instead of finger to MPC pad. He’d rap, just like he had done with Detroit trio Slum Village in the 1990s. And the beats would be from producers he admired and respected. Beats, rhymes, and hype.

Dilla’s MCA album became the stuff rap dreams are made from, a myth of its own.

The project began in earnest at the turn of the decade. In Detroit, where winters are harsh, bedroom studios are most often basement hideouts and Yancey had the best of them all. Nicknamed The Spaceship by friends, thanks to its gigantic, state of the art mixing desk, it was located in Clinton Township, about 20 miles northeast of Detroit. That’s where Yancey handled pre-production for the album. Beats were procured from Pete Rock, Madlib, Hi-Tek, Houseshoes, Waajeed, and a young Kanye West, among others. Demos were recorded at Studio A, in nearby Dearborn. Frank N Dank and Bilal dropped verses and sang hooks. Revisions were made after everyone had left the studio. Demo CDs were sent to the label outlining potential tracklists. And then the album was shelved. Wendy Goldstein, senior VP of A&R at MCA and Yancey’s champion, left for Capitol Records. Her projects became burdens and Yancey was once again sidelined by the industry. The MCA album became the stuff rap dreams are made from, a myth of its own.

Yancey moved to California in the spring of 2004 after his basement studio flooded and the MCA deal soured, seeking new prospects at the edge of the western world, where countless explorers had gone before him. Within a year he was regularly in and out of hospital fighting an incurable blood disease, thrombotic thrombocytopenic purpura. On February 10, 2006, three days after his 32nd birthday, he died. Even before those who knew him could mourn him, everyone began to claim the man as their own. In the decade since Yancey’s death, his oeuvre has been dissected with such intensity that there remains little to slide under the microscope – except the MCA album.

Now called The Diary, the lost album will finally see light of day on April 15, released under the artist name J Dilla. No major labels, no bootlegs, no bullshit: just the singular vision of one of hip-hop’s greatest pioneers given shape by a handful who knew Yancey well and knew what the album meant to him. In many ways, the story of The Diary resembles the story of Yancey’s life: it wasn’t easy, but it was hopeful.

“This is the last record he wanted out.” Eothen “Egon” Alapatt

Yancey’s illness, and the suddenness with which it hit, left him unable to ensure his life’s work would be well overseen. On the night of September 8, 2005, when it looked like he might not make it till morning, Yancey signed a will. Most of those around him did not know about it, though he’d once discussed the idea with Joylette Hunter, mother to one of his daughters. Brought in by his then-attorney Micheline Levine, the will named his brother, John “Illa J” Yancey, his mother, Maureen “Ma Dukes” Yancey, and his daughters, Ja’Mya Yancey and Ty-Monae Whitlow, as heirs to his estate. Artie Erck, Yancey’s accountant, was named as the estate’s executor. The Detroit producer was the first in his family to have anything of worth to bequeath. As his life grew darker he put pen to paper, not knowing that the will would interrupt his legacy rather than solidify it.

Eothen “Egon” Alapatt did not know about the will. What he knew was what Yancey had told him before he died. “This is the last record he wanted out,” Alapatt recalls, speaking from his office in northeast Los Angeles. “That to me is as important as anything. If you believe in Dilla and he wanted this out, or even if you have the most passing interest in who he was, you should defer to him and give this record a listen.”

Alapatt is an energetic and enthusiastic speaker, his love for music and Yancey apparent in his every word. He first discovered Jay Dee on the credits of The Pharcyde’s 1995 single ‘Runnin’ and decided he “had to know everything this guy did.” He began working with Yancey as general manager of LA-based label Stones Throw in the early 2000s, after Otis “Madlib” Jackson Jr. was invited to work on the MCA album. From there, Alapatt and Yancey became friends, their relationship solidified through work, spiritual kinship, and, finally, death. In early 2007, Levine called Alapatt to offer him the position of creative director of Yancey’s estate. The estate was to utilise the intellectual property and work he had left behind to repay his large debt to the IRS, accrued in part due to his illness and lack of medical insurance. After that, probate (the legal process by which a will is proved in court) would close and the estate would be handed to the heirs. But Yancey’s business dealings had never been simple in life, and that bad luck dogged him in death.

At first, many assumed Yancey’s mother was the executor of his estate. Instead, Alapatt was told that no money from his work could be given to her directly unless it was a gift. By then, “Donuts had gone from being an idea to very profitable,” he says. The record had saved Stones Throw, which was teetering on the brink of bankruptcy. There was money to disburse to the heirs, but doing so through the estate would lock it up. Compounding matters, the process of caring for Yancey had left Maureen broke, and it was becoming clear that Erck, the accountant and executor, had no intention to help. As general manager of the label, Alapatt made the only choice he could. “Erck had an adversarial relationship with Maureen, [so] Stones Throw continued making payments to her because Jay’s allegiance had made us profitable and we believed in filling the void left by his death,” he says. “I ran the company day to day. I cut the cheques. It was something I believed we should do, and we did it.”

One of the first things Alapatt looked into as creative director was the possibility of releasing the lost MCA album, which he’d begun working on the month after Yancey died by securing the master tapes through Maureen. Erck confirmed that legal documents had freed Yancey from his label contract, allowing the estate to use the album. As work continued, Alapatt realised Erck was more “busy thinking about how he and his cronies could make money” from Yancey than building a strong legacy. He remembers how “Artie, Micheline and those people were at the complete opposite end of the understanding Dilla had hoped they would have.” This new and precarious relationship soon came to a head after Alapatt asked the wrong questions about payments to the heirs and Erck and Levine tried to push him out of his role. A legal battle ensued, culminating in February 2009 with a victory for Alapatt and the heirs. Erck left his position and a new independent executor, Alex Borden, was appointed by the state of California.

But while lawyers did the dirty deeds they are handsomely paid to do, life carried on. February, the month of Yancey’s birth and death, had become a yearly focus for worldwide tributes and celebrations. Maureen, shut out of the estate, set up the J. Dilla Foundation as a non-profit organisation to help fund progressive music education programs. In April 2008, an unfinished version of the MCA album was leaked online and bootlegged on vinyl under the name Pay Jay, leading Erck to publish an ad in Billboard that summer stating he was the only person able to authorise use of Yancey’s intellectual property. By early 2011, when Alapatt returned to his position as creative director, there’d already been two posthumous albums. Momentum had been lost and Dilla’s legacy was a mess.

The goal for Alapatt, helped by the estate’s new attorney Sheila Bowers, who had worked with Stones Throw and Yancey before, became simple: set up a functioning, profitable company to warehouse all of Dilla’s intellectual property, so that when probate closed it could be given to his heirs. That company was Pay Jay.

“I couldn’t be happier with the result – to me it sounds like what Dilla would have done.” Eothen “Egon” Alapatt

When Alapatt returned to work on The Diary he did so a different man. Past the legal travails, he’d also left Stones Throw and was now dedicated to his Now Again label, home to historical work and reissues, and to Madlib, whom he’d managed since the early 2000s. In 2010, Alapatt and Madlib began to imagine a new type of label based around a singular artistic vision with Madlib Invazion. “We’re at a point where the sustainability of a record company at a large level is in question, but the idea that artists like Madlib can sustain a career and release music is possible,” he explains, extending the analogy to Yancey’s legacy. In many ways, Madlib Invazion offered a template for Pay Jay. “I could see it was possible to do it on our own without any label infrastructure by outsourcing everything and hiring the right people on a per-project basis. There are projects that need extra push and those that don’t, such as those done for spiritual reasons because that’s how Madlib and Dilla make music. Music needs to go into the world.”

Used to making sense of the fuzzy genius of Madlib, the Sun Ra to Yancey’s Coltrane, Alapatt was in the perfect position to piece together the puzzle of The Diary. As the team blew the dust off the master tapes there was a lot to unpack, beginning with files bundled in archaic software packages. The person with the key to the vault was Dave Cooley, an old friend of Alapatt who runs the Elysian Masters studio in Los Angeles. Cooley and Yancey “got along famously”, and he had been one of Yancey’s go-to engineers before his death. Unlike the many rock and pop professionals who populated LA’s studio landscape at the time, he didn’t question the coherence of an album but instead turned in the best possible sounding record, regardless of how much he had to work with. Legend has it that Yancey had a voicemail message set up in case people came requesting multi track versions of his beats, with separate sources for the various sonic elements. “Two-track that shit,” the message said, referring to the final stereo two-track mix overseen by Yancey. You didn’t question a man known for impeccable sonic accuracy.

Once Cooley had pried the files from the master tapes the puzzle was laid out in full, with dozens of folders outlining revisions, variations, and unfinished mixes of the album’s tracks. J.Rocc, Yancey’s tour DJ, lent his expertise to organise the files into the most likely order, while Cooley finished the job of mixing down and mastering the songs. “Dave was the obvious choice to figure out the mixdowns because he views this empirically,” Alapatt says. “I couldn’t be happier with the result – to me it sounds like what Dilla would have done.”

When working on an archival project without the artist around “the most important thing to verify is intent,” says Alapatt. What did they want? And how close can we get to it? To do this requires working with those who knew the artist and who worked on the project. Alapatt was well placed to do this, thanks to his relationship with Yancey and reissue experience through Now Again. “At a certain point a muscle memory develops,” he says. “I tried to have a dialogue with as many of the people involved as I could, but I knew the final decision was mine, so it wouldn’t be by committee. I felt I was tasked with this. Even the collaborators had to admit they weren’t there for the final versions. We had to choose between the commonly known song in the narrative of this album and the revision Dilla probably would have wanted on there.”

Kanye West’s contribution, revealed through the Pay Jay leak, was one such example. West didn’t know Yancey had recorded over the beat, a gritty draft of what became Jay-Z’s ‘Takeover’ in 2001. In the end, Yancey produced the song himself as ‘The Anthem’. It’s the entire opposite of the version we thought we knew. A slippery groove runs drunken rounds around a string sample and the energy switches from aggressive to euphoric.

Alapatt calls the record “85, 90%” accurate to what Yancey would have wanted. Michael “Houseshoes” Buchanan, a long-standing collaborator who was in Detroit during the original recording sessions, says that The Diary is “the most accurate shit we’ve heard since Jay left. And accuracy is the most important thing when dealing with someone lauded for being so critical of the sonic quality of his music.”

The lead-up to the album began in 2013 with the first official single on Pay Jay, which included ‘Trucks’, Yancey’s reinterpretation of Gary Numan’s ‘Cars’. The night ‘Trucks’ was completed, Houseshoes got a call to roll through Studio A at 3am. “He was so fucking excited, man,” the local DJ recalls, “like he finally had the record.” Another three singles on Pay Jay followed, unveiling further finished tracks and B-sides, leading to the announcement of The Diary in February this year. The reveal came with a new song, ‘The Introduction’, produced by Houseshoes and Yancey. It’s an ominous album opener with hand-played synthesiser arpeggios and stabs laying a corrugated base for Yancey to flip the opening lines to A Tribe Called Quest’s ‘Excursions’, a tip of the hat to his past. He then proceeds to tell us what he’s about to do on the album in one brutal verse. “It’s breathtaking,” Alapatt says. “Listening to it, I couldn’t wait until people heard this. I had a few of those moments in the process of making the record.”

Also fascinating is the closing title track, with Yancey retelling parts of his personal and family history over nostalgic strings and the stinging chorus, “You think you know / you have no idea.” This album is, in the words of Houseshoes, “the last piece of the puzzle.”

“He wasn’t the guy to say ‘black lives matter’, he said ‘fuck the police’. Two different ways of saying the same thing.” Eothen “Egon” Alapatt

The Diary is the most comprehensive document of Yancey’s duality as a rapper and producer. While he is best remembered for the latter, to dismiss him for the former was always a grave mistake. Yancey “knew he was a bad motherfucker” says Houseshoes, a fact apparent in the majority of the rhymes he left behind. But the braggadocio was just one aspect of it. Producers also hear a beat like no one else. Alapatt brings up Madlib’s rapping alter ego Quasimoto as a case in point. “If you’re focusing on why he’s not a great rapper, you’re missing the point of him rapping,” he says with a laugh. “Rapping producers feel the music differently.”

Today, being a rapper and producer—and maybe even a designer and filmmaker—is a given. But it took decades for such habits to sink in. During Yancey’s life, the only way out of the shadows for a producer was to rap. On The Diary, Yancey transforms from the quiet and humble figure hidden in the studio into a proud lyricist, the same one who can be seen pacing stages in old footage of Slum Village and Jaylib. He talked about being the best, but also about “love, lust, longing, depression and the exhilarating feeling of purchasing something if you never had anything,” like the jewellery and cars he loved so much. Those are stories about Yancey we haven’t always told.

One song on The Diary still resonates more than most today. ‘Fuck The Police’ was recorded in the early 2000s following a late night police stop on a food run while recording at his Detroit studio. Yancey was driving a Range Rover. Nothing flashy, but enough to warrant the attention of overbearing police officers in a broken city of a broken country where a broken system of law has for centuries viewed black men as a potential danger to be curtailed, perhaps even erased. “The cops were belittling him and it hurt him so bad,” said his mother in 2008 of the incident. She suggested he channel his anger into music. The result is one of the most scorching denunciations of police brutality since N.W.A, the power of his words laid bare over a simple drum and flute loop from a little known library record. One line retains a poignant urgency: “See now they got these fake cops / Thought he had a gun / Made a mistake cop.” The same fake cops gave rise to protest groups like The Dream Defenders and Black Lives Matter in recent years. “Dilla didn’t beat around the bush,” says Alapatt of the song. “He wasn’t the guy to say ‘black lives matter’, he said ‘fuck the police’. Two different ways of saying the same thing. One is less abrasive but the other is no less telling.”

Just as Yancey’s lyrics were brash, his beats remained subtle. This opposition was the root of his artistic duality. As a decade of documentation attests, Yancey made his music say what others in hip-hop had never thought of. Even after 10 years, we’re still finding new meanings between the bars and inside the loops. It’s therefore no surprise that Kanye West, who’s today at the centre of the discussion of what being a rapper-producer means, has called Yancey “the greatest to do it,” as he told Madlib during the recording of his last album.

The Diary is a stunning sonic fossil, a vestige of a time past yet still able to inspire awe.

Through the hard work of Alapatt and the estate, one of Yancey’s final wishes is now realised. The Diary is, by its very nature, a time capsule returning us to a transitory moment when hip-hop’s underground realness was giving way to mainstream flash. There are dated lyrical references and some of the beats Yancey sourced are clear markers on the ongoing timeline of hip-hop production, like Nottz’s stuttering chops on ‘The Shining Pt. 1’ or the sped-up vocals and uplifting strings of ‘So Far’ by Supa Dave West. The album’s freeze-frame of Yancey as he was between 2000 and 2004 also reveals previously hidden links between phases of his career and styles he came to champion. ‘The Anthem’ and ‘The Shining Pt. 2’ could be taken from Jaylib, and ‘Give Them What They Want’ sounds like a lost Ruff Draft cut, two albums born from the frustrations of the failed MCA deal. And because he’d come to change the game, the album also has a timeless quality. It is a stunning sonic fossil, a vestige of a time past yet still able to inspire awe.

That The Diary sees release a few months after the 10th anniversary of Yancey’s passing is fortuitous. A cosmic wink from the ether, a hopeful touch to a sad story. At last “the lines are clear.” The estate of James Yancey is now in the hands of an impartial executor whose only interest is to see probate close. Unlike all other posthumous projects to date, The Diary is a literal interpretation of the creative vision Yancey could wield. “It feels triumphant,” says Alapatt, “but also really heavy. Because this was the last thing he wanted out. After this…” His voice trails off. “Talking of revisionism,” he adds with a nervous laugh.

There’s another story. The version of Donuts we all know wasn’t the one Yancey had handed to Alapatt after he finished the album. The Donuts we’ve heard was edited into a longer form by Jeff Jank, Stones Throw’s art director, who convinced Yancey to let him change it. And while Yancey was canonised and Donuts became his final statement, a different masterpiece sat in a closet for years, burnt on CD. This original take is shorter, with different sequences, edits, and transitions. The current idea – and it is only an idea – would be to allow for this different batch of donuts to be released via Pay Jay so it can benefit the heirs. “I think there’s a creative vision Dilla had for this record which would be interesting to put out into the universe,” says Alapatt.

We tell ourselves stories because we need something to believe, and because we crave order and understanding in our constructed personal realities. Which is why there’s something beautiful in the knowledge that after 10 years and countless stories, Yancey is still capable of surprising us.