Andrew Crofts has written 80 titles and sold some 10 million copies in a 40-year career, mostly under names far more famous than his own – until now

"Behind the title of ghostwriter, I could converse with kings and billionaires as easily as whores and the homeless, go backstage with rock stars and actors. I could stick my nose into everyone else's business and ask all the impertinent questions I wanted to. At the same time, I could also live the pleasant life of a writer… "

Next week, in an exceedingly rare departure from a lifetime of tight-lipped professional discretion, Andrew Crofts, one of Britain's most invisible and yet successful writers, a bestseller you will never have heard of, will step out of the shadows and lift the veil on a trade that's almost as old as that other ancient calling. With a bit of skirt-lifting, and more than a hint of saucy revelations, Confessions of a Ghostwriter will be a timely publication.

There's an old saying that you should never judge a book by its cover. Today, perhaps, that conventional wisdom has rarely had more meaning. To a degree that might astonish the reading public, a significant percentage of any current bestseller list will not have been written by the authors whose names appear on the jackets.

Among the many mysteries of the British book world, none is quite so opaque as the life of the ghostwriter, the invisible man or woman who fulfils the vanity of those who want their name on the cover of a book but who, for the life of them, cannot write.

You may not know it, but literary ghosts are everywhere. In this golden age of reading, publishers desperate for copper-bottomed commercial titles in bestselling genres – misery memoir, sporting lives and celebrity autobiography – will not hesitate to sign up surrogate authors.

Behind such brand names as Sir Alex Ferguson, Jordan, Andy McNab and Victoria Beckham lurks the shy figure of the ghost. Sometimes, there is no deception. Keith Richards's Life was written by James Fox. Katie Price (aka Jordan) boasts that she does not do her own typing, and relied on Rebecca Farnworth to launch her career as a novelist with Angel. Further down the food chain, even the infuriating meerkat from the comparethemarket.com adverts has had A Simples Life put together by Val Hudson, formerly of Headline books.

The top category of ghosted titles remains the misery memoir, books such as Tell Me Why, Mummy or Please, Daddy, No, or Sharon Osbourne's Extreme: My Autobiography. At its peak, this genre accounted for almost 10% of the UK book market, closely followed by celebrity autobiographies (Russell Brand's My Booky Wook), true-crime memoirs (Dave Courtney's Stop the Ride, I Want to Get Off), sporting lives (Wayne Rooney's My Story So Far) and tales of derring-do (Bruce Parry, Bear Grylls, et al).

Ghostwriting in the English-speaking world is big business. The term was coined by an American, Christy Walsh, who set up the Christy Walsh Syndicate in 1921 to exploit the literary output of America's sporting heroes. Walsh not only commissioned his ghosts, he imposed a strict code of conduct on their pallid lives. Rule one: "Don't insult the intelligence of the public by claiming these men write their own stuff."

Walsh's code lingers. The acknowledgments page of many ghosted books will thank partners, children, even family pets, before making a discreet, sometimes grudging, nod to the invisible man or woman who quarried the angel from the marble. Alternatively, and more transparently, the book will be credited "as told to", or "written with", or "edited by".

Those innocuous phrases often mask a world of private pain: tearful interviews, angry confrontations, threats of violence, shocking revelations and interminable waiting, waiting, waiting. In France, ghosts are known as nègres, and there is a kind of slavery implicit in this transaction. The ghost's world may be one of jeopardy, but it's probably less perilous than it is depicted in Robert Harris's thriller The Ghost, a book credited by many with outing the ghost's tradecraft.

As with any book, the struggles of the ghosted book are all to do with love and money. First, there's the inevitable contract tussle. Traditionally, the ghost receives 33% of the advance (plus royalties). In recession, this has been squeezed to as little as 10%, a figure the better class of ghost will disdain.

Often, battles over the money pale into insignificance next to the titanic clash of egos involved in taking on another's voice and character.Some ghosts, who generally speak on conditions of anonymity, report that the subject they approach with utter dread is the fragile personality with pretensions to authorship.

Who, after all, is not vulnerable to the tug of amour-propre? The ghost, who starts out as a hybrid of therapist, muse and friend, enters a psychological minefield. Accordingly, the ghost is advised never to forget that, at the end of the day, he or she ranks somewhere between a valet and a cleaner.

I recall, some years ago, a female pop star attending a book trade prize-giving for which her ghosted bestselling memoir had been shortlisted. Before this honour, she boasted she hadn't even opened, still less actually read, the book that bore her name. When she duly won, she left her ghost at the table and graciously collected her prize, all smiles, modesty and gratitude, the model author. When she returned to her publisher's table, the woman who had actually written the book reached out, instinctively, to touch the trophy. Bad move. The star snatched it back, clouting her ghost across the cheek to remind her who was boss. When you pay the piper, you call the tune.

Crofts has written some 80 books, sold more than 10m copies and appeared a dozen times in UK bestseller lists. In a rare interview with the Observer, Crofts described some 40 years of ghosting. An easygoing, youthful man in his early 60s, Crofts was educated at Lancing College, but says he was "too arrogant" for university, and stumbled into ghostwriting because, he says, "I didn't want to have a permanent job".

Ghosts, notes Crofts, lead episodic lives: "It's a perfect arrangement. You get the commission, have the adventure – anywhere from a palace to a brothel – and return to the security of your own home." Crofts is a child of the 1960s who seems to have transformed a secret vagrancy into a way of life. At 17, on leaving school, he nurtured vague literary ambitions, wrote a novel ("more Robert Harris than Virginia Woolf"), suffered the inevitable rejection and began writing PR copy. With typical English self-deprecation, he describes himself as "an opportunistic hack" who would "do anything I got paid for". When pressed, however, he admits to taking pride in his craftsmanship and in having made "a good living as a writer for 40 years".

When he started, he recalls, "'ghostwriting' was a dirty word". In 1984, with the chutzpah of youth, he launched himself in business. His approach was simple and direct. He placed a three-word ad – "Ghostwriter for hire" – in The Bookseller, and waited for the phone to ring.

Crofts was lucky, with impeccable timing. Book publishing would be turned inside out by the IT revolution. Ghostwriting, similarly, was transformed by the web. "The internet made all the difference", says Crofts, who was one of the first ghosts to launch his own website. Now, he gets three or four approaches a day. "I'm writing all the time," he says.

Under his own name, and from a certain pride in his trade, he went on to publish Ghostwriting, a how-to manual. When Robert Harris read this as part of his research for The Ghost, he sought permission to quote some of Crofts's obiter dicta ("Of all the advantages that ghosting offers, one of the greatest must be the opportunity to meet people of interest") as chapter-heads. The Ghost, says Crofts, was "a gift from the gods. Harris did us all a huge favour."

Since 2007, Crofts has become the ghost's ghost, the go-to spook in a now-booming market. "I charge a lot," he admits, and concedes that his fees average six figures. Crofts, who currently earns more than most professional UK writers, is sought after by overseas celebrities, politicians and stars, especially in India. He also works with Russians, Africans, Arabs, and South Americans. "Everyone loves London," he says. "This is soft power in action. London is seen as the home of publishing, a place that's kosher, where Dickens walked the streets."

His rule for accepting a new client is that they must have a good backstory. He took on Alexandra Burke (of The X-Factor) because of her mother's career sacrifice. "She was in hospital watching her daughter on TV, living her life," he says, and confides a special interest in tales of childhood abuse. Sold, the shocking rape story of Zana Muhsen, has shifted 5m copies and, Crofts believes, created a new market for books such as Jane Elliott's The Little Prisoner. Crofts also took on Big Brother's Pete Bennett, an acute Tourette's sufferer, out of respect for "an extraordinarily attractive character", and ghosted Pete: My Story, another bestseller.

Is there anything he wouldn't do? "I have to be interested", he says, conceding that he could happily coexist with monsters. "I have a horrible feeling that if I'd got the call from Germany in the 1930s I would have hopped on that plane like a Mitford."