When he was a teenager growing up in Phoenix, Maryland, Dan Piepenbring would often head to Walmart to check out the new release CDs. It was 1999 and the height of rap rock and nu metal, of Limp Bizkit and Korn, of hirsute man-boys with snapback hats and chain wallets. Piepenbring’s curiosity was aroused by something else – a record called Rave Un2 The Joy Fantastic. “The singer was wearing a blue shiny outfit,” he recalls. “Not quite superhuman, but extraterrestrial. He had no name and looked so effeminate. He seemed to be of indeterminate ethnic origin. And I was a little afraid. He just seemed so weird. I remember handling the CD on successive visits to the store and thinking, ‘What is this?’ It was so gloriously out of sync with everything else on the shelves.”

That was Piepenbring’s first encounter with Prince. In 2004, he listened to Purple Rain and was smitten. “I became obsessed,” he says over hamburger and fries in a Greenwich Village bar. Now 33, he has the underfed eloquence of the eternal postgrad. “I had this sense that I was listening to someone who combined so many fascinating strains of popular music, from the last 50 years, in an original package.” In 2016, Piepenbring was working as an editor at the New York-based literary journal Paris Review, when his agent told him Prince was searching for a co-writer to work on his memoir. He turned in a pitch-cum-love-letter: “From my first encounter with Prince I knew he was a master storyteller,” he wrote. “To help him tell his stories in a new mode would be a once-in-a-lifetime honour.”

Less than a day later, he was summoned to meet the singer at his Paisley Park complex in Minnesota. He got the assignment. What followed he describes as a “bizarre, three-month detour in my life, a strange and voluptuous period. And so surreal.”

Piepenbring and Prince went to concerts, parties and a private screening of Kung Fu Panda 3 together

Piepenbring was a surprising choice, being all the things Prince was not: white, young, college-educated, outside the musical world, completely unknown. But Prince was famously secretive about his life, as well as dismissive of interviewers. He described writers as “mamma-jammas wearing glasses and an alligator shirt behind a typewriter” and claimed, “There’s nothing a critic can tell me that I can learn from.” Piepenbring remembers him saying that he regarded most of what had been written about him as “garbage”. The Paris Review is best known for its extended interviews with authors, but this did not stop the young editor becoming Prince’s amenuensis. “My first Monday coming back from Paisley Park, we gave over the weekly staff meeting to having me retell the story of that weekend. Then I was like: if any of this gets out I will never get this job!”

Facebook Twitter Pinterest ‘Hey, did you know that too much sex makes your hair grow?’; a high school comic strip stored at Paisley Park. Photograph: Copyright The Prince Estate

Over the next few weeks, Piepenbring and Prince went to concerts, parties and a private screening of Kung Fu Panda 3 together. They had conversations over the phone and in Prince’s black Lincoln MKT (his driving, particularly his turn signalling, was excellent). The singer mentioned how at high school he’d punched a kid who’d thrown a racial slur at him – and that his best friend growing up was Jewish. He spoke about the death of an early love – the singer Vanity. “He had a great, deadpan and warm sense of humour. I was surprised by his autonomy and his ability to put people around him, including me, at ease.”

The writer flew to Australia where Prince, who was checking into hotels under the name Peter Bravestrong, discussed with him his ideas for the book. “Is it about the music industry? His parents? A handbook for the black community? His ideas could change in the course of a single conversation. I don’t know which one he would have settled on,” he says.

A degree of trust had been established – Prince referred to Piepenbring as “my brother Dan” – and the singer talked of having the pair’s distinct voices merge over the course of the narrative. Thinking back to the epilepsy he suffered as a child, Prince suggested, “We could use seizures as a way of blending our voices. Blackouts.” I tell Piepenbring this sounds more like avant-garde fiction than a traditional rock memoir. “Which I was so looking forward to! The fact that he wanted to produce something so formalist and postmodern and strange was the most surprising thing. Especially because, at this point in his career, he was writing songs in various generic modes rather than, as he did in the 1980s, shattering those modes. But what a burden! There’s no guarantee it would have been readable, or necessarily true to what he had imagined.”

Facebook Twitter Pinterest One of a series of shots Prince took in the bathroom mirror of a rented house in the Hollywood Hills, Los Angeles, 1978. Photograph: Copyright The Prince Estate

On 21 April, barely three months after they had started collaborating, Prince’s death was announced; he had overdosed on fentanyl. The last time the two men had spoken was four days earlier, when Prince had phoned, talked a lot about his parents’ influence on his life, ending: “I just wanted to call and let you know that’s what I’ve been thinking about. And I’m OK.” Was he lying? Was he, in fact, in great pain? “My thinking on this changes hour to hour. There were moments when I was certain he must have known, because to do those stripped-down Piano & A Microphone shows, and to pair that with an autobiography seems so summative. It’s rooted in a past that for the longest time he would have eschewed. Prince may have known he was in poor health, but there’s a difference between that and knowing he was dying.”

For someone who said he didn’t like to dwell on the past, it seems like he was hanging on to everything

His death seemed to spell the end for The Beautiful Ones (named after a song on the Purple Rain album), whose imminent publication had delighted fans. “After he died, I was a mess,” Piepenbring says. “I was cranky, depressed. And I also felt a fool for feeling depressed: I didn’t really even know this guy that well. But often I felt hurt and deprived and confused and angry at myself. And sometimes even at him.”

Facebook Twitter Pinterest A page from a photo album Prince put together when he was 19 and recording his debut album, For You, in California. These shots were taken by his manager, Owen Husney, in July 1977. Photograph: Copyright The Prince Estate

Prince died without making a will and in the months following his death his estate was run from the courts, with advisers coming and going, claims from various family members, constant threats of litigation. “A book wasn’t on the top of anyone’s mind,” says Piepenbring. “They needed to bring in a lot of income, and quickly. They were probably thinking: which album can we release? What’s in the vaults?” Yet he couldn’t forget his time at Paisley Park, the exchanges and intimacies he’d shared, and started to think of resurrecting the book, now partially reconceptualised as one with “a negative space at its centre, around whose mystery I orbit.”

With his editor and publisher, Piepenbring gained access to the fabled vaults at Paisley Park. Prince fans, aware of how much music their hero withheld from the world, tend to regard this archive much as Howard Carter must have dreamed of Tutankhamun’s tomb. “We were struck by the sheer breadth and volume of it,” Piepenbring says. “For someone who spent much of his career saying he didn’t like to dwell on the past, it seems like he was hanging on to everything – wedding programmes, wedding gifts, a set of his-and-hers bowling balls (the latter engraved with the Prince squiggle and the name of his wife, Mayte).

Facebook Twitter Pinterest By late high school, Prince wrote, he had bought a ‘vanilla stratocaster identical 2 the one Jimi played at Woodstock’. Photograph: Copyright The Prince Estate

“The sheer quantity of paper was surprising. There was no real method to the madness. In one room you’d find something from 1979, and within arm’s reach there’d be something from 2002. There’d be these moments of intense intentionality – he’d gathered up all his handwritten lyrics from across his career, clipped them together, and kept them in one place. Or he’d gathered artefacts pertaining to his father’s jazz band. It seemed very ad hoc and very personal.”

The estate, which is currently run from the Carver County courts in Minnesota, limited access to the physical vault, pre-curating the material by digitising sections of the archive, which were uploaded on to a dedicated site from which Piepenbring selected material. So he hasn’t seen everything. “It’s Rumsfeldian,” he laughs. “There are known unknowns, and there are unknown unknowns.” Still, responding to some of the words Prince repeatedly came back to during their conversations (freedom, creativity, community) he has fashioned a riveting anthology that includes many unseen photographs of Prince as a teenager, handwritten lyrics, and photographic contact sheets that illuminate the creative guile the singer put into finessing the persona that shocked and seduced the general public.

They keep trying to ram Katy Perry and Ed Sheeran down our throats, and we don’t like it," Prince wrote

“There are thousands of staged photoshoots of Prince,” Piepenbring observes. “I wanted things that gave me the same frisson as when we met. I could hardly believe I was seeing a mirror selfie of Prince. And that he went on taking them through his life – almost as a way of checking in on himself. I also love that photo of him with a guitar: it’s the two objects that most readily come to mind when you think of Prince – guitar and bed – in a perfect combination of virtuosity and intimacy. He’s got this very intense but abstract and insouciant look on his face. He seems almost lost in thought, and there’s a bit of melancholy in there, too. It’s such a lovely image.” Asked why he has included so few photos from after 1986 in the book, Piepenbring argues not that Prince was too busy to maintain his archive but that, by then, “Prince is in such professional hands that the choreography is complete.”

The book contains cartoons by Prince (he called them “Prince’s Funnies”), details of his first kiss, memories of watching R-rated movies at the local drive-in, and disses of the media conglomerates who control contemporary US music (“We need to tell them that they keep trying to ram Katy Perry and Ed Sheeran down our throats and we don’t like it no matter how many times they play it”). Of his mother, he recalls, “She would spend up what little $ the family had 4 survival on partying with her friends, then trespass in2 my bedroom, ‘borrow’ my personal $ that N’d gotten from babysitting local kids, & then chastise me 4 even questioning her regarding the broken promise she made 2 pay me back.”

Facebook Twitter Pinterest Outtakes from the cover shoot for Parade, released in 1986. When Prince renewed his passport a few months later, he cropped a headshot from one of photographer Jeff Katz’s images. Photograph: Copyright The Prince Estate

His self-possession seems always to have been there: “Kids & teachers would tease me about My name but it never bothered Me because it was unique. No One Else had the given name Prince.” He developed his performance philosophy early, too: “A vibrant imagination is where the best songs R found. Make-believe characters wearing make-believe clothes all 2gether creating memories & calling it Life.”

One of the most revealing features of the book is its reproductions of Prince’s handwriting. “It was the perfect physical emblem of this tension in his life,” says Piepenbring. “The beautiful, finely wrought scripting is also pouring out of him so quickly that it borders on illegibility. On the one hand, discipline; on the other, free-spiritedness.” Why, I ask, did he insist on using the letter ‘N’ for ‘I’ and ‘U’ for You’? “I would have wanted to ask him about glyphs. Was it disorientation? A private language? Something more playful? He was fascinated with codes and told me of feeling that his childhood home was encoded: he would look at the wallpaper and certain objects and feel there was a message in them.”

This year, limited-edition and previously unreleased Prince recordings are starting to be issued; his catalogue has been made available for streaming. But Piepenbring finds it hard to visit Paisley Park itself. “The museum there is a wonderful experience, but you can feel that something has gone. There’s nothing that compares with Prince being there when he was alive. There used to be a sense of a flame being guarded.”

Facebook Twitter Pinterest In bed on France Avenue, Edina, Minnesota, April 1978. Prince’s first home, it was bought with a Warner Bros advance, and became the site of his first DIY basement studio. Photograph: Copyright The Prince Estate

The book is not the one Piepenbring envisaged. Our conversation is full of sighs and absences, spectres and conjectures; Prince, so cryptic in his lifetime, remains elusive. How does his ghostwriter feel now, looking back on those intense three months?

“I’ve had dreams about Prince, where he’s still alive and his death was just a widely misreported thing,” he says. “I’m still reluctant to handle those memories. They’re almost like something in a vitrine: the more you look, the more you get hot breath and fingerprints on the glass. Suddenly this very pure thing is smudged and you can never see it as clearly again.”

• The Beautiful Ones by Prince is published by Cornerstone, at £25. To order a copy for £22, go to guardianbookshop.com or call 020 3176 3837. Free UK p&p on all online orders over £15.

Dan Piepenbring will be in conversation at The Beautiful Ones Live on 4 November at EartH, east London. Tickets available at penguin.co.uk/events.

If you would like a comment on this piece to be considered for inclusion on Weekend magazine’s letters page in print, please email weekend@theguardian.com, including your name and address (not for publication).