Nigel Hamilton opens his new primer How to Do Biography (Harvard) with the bold boast that we are living in "a golden age" of life writing. Really, he should know better. To anyone who reads, reviews or writes on the subject, such confidence is baffling. (Hamilton, a Briton, lives mainly in the States, which may account for his rosy myopia.) Seen close up, and with an eye to proper detail, biography appears in rather a bad way. "Crisis" would probably be putting it too strongly, not least because it suggests a certain convulsive energy. "Sclerosis" might be nearer.

Sales, it's true, are still good, though showing signs of softening. According to Nielsen BookScan, literary biography reached an all-time high in 2005, but has since started to fall. General arts biographies are also down. However, to give an idea of how the non-fiction market as a whole has recently been bent out of shape, it's worth noting the exponential leap in celebrity memoir. Thus Katie Price has managed to shift 335,649 hardback copies of her life story Being Jordan, despite her jaunty admission that someone else wrote it. Meanwhile, Hilary Spurling's Costa-winning Matisse the Master, surely one of the best biographies of the decade, has lifetime hardback sales of just 12,451.

However, it is when you look at the quality of work produced rather than the number of books sold that you start to fear for the health of a genre that not only predates the novel by centuries (think of Plutarch's Lives), but holds peculiarly British credentials. Since becoming a biographer 15 years ago - most recently of Mrs Beeton and George Eliot - I have read widely within the genre and with professional attention. I also teach on the MA in life writing at UEA, which aims to give would-be biographers a solid grounding in their chosen art. And I get hundreds of titles sent to me every year for review. I have to say I am struck by their recent falling-off. Of course there are always stand-outs, even in a thin year such as 2007 (Hermione Lee's Edith Wharton and Rosemary Hill's Pugin), and 2008 is shaping up to be better, with new work from Richard Holmes and Michael Holroyd in the autumn. But the general standard, the mean, the middling, seems to have sunk to a listless low.

Let me give you a taste of what I am sent each week. There are endless lives of saucy 18th-century society ladies, even though it is a full 10 years since the surprise hit Georgiana by Amanda Foreman opened up that particular realm. (The worst moment was receiving a biography of Georgiana's sister, Harriet.) There are plenty of royal mistresses, to the extent that one wonders if there is a boudoir somewhere where they sit all day doing their hair, waiting to be called forward for active biographical duty. Lives on the edge of empire are popular too, probably in response to the academy's continuing interest in all things postcolonial. So each week you can be guaranteed stories about people who ran little kingdoms in clammy parts of the globe and/or were worshipped as gods by confused locals.

While there is a respectable rationale for doing these minor lives, grounded in the argument that hidden or forgotten subjects deserve their restorative moment in the sun, the real reason is more pragmatic. The fact is that anyone really important has already been done, probably several times. The 19th and 20th centuries have long been harvested for any royal, writer, actor, painter or soldier whose life and work could conceivably yield 350 pages of serviceable prose. (That's another problem: the misguided belief that writing a biography requires nothing more than journeyman skill when it comes to putting one word in front of another.) For a while, the 18th century appeared fertile, with its combination of ebullient consumerism and unbuttoned sexuality seeming to offer an easy parallel to our own. Now, though, it is all fished out. Go back any further than 1700 and you run into the problem not just of finding the sources, but of making sense of them. People in the early modern period had such different ideas about what it meant to be a self, especially in relation to others, that trying to corral them into the kind of unfolding psychodrama expected by the modern biography reader becomes tricky, unless you're prepared to gloss like crazy.

The least imaginative response to this lack of good new subjects is simply to go back to the big lives and do them over - and over - again. You can justify this by an appeal to the idea that each decade (actually, every four years might be nearer) needs its own Dickens or Eleanor of Aquitaine. According to this convenient way of thinking, dressing up old subjects in new clothes becomes playful and postmodern, rather than just desperate. Yet Jerry Brotton of Queen Mary, London University, whose The Sale of the Late King's Goods: Charles I and His Art Collection reached the Samuel Johnson shortlist two years ago, warns against being taken in by this sleight of hand. "What often happens is that a perfectly solid biography from 1978 gets rewritten with an eye to the intellectual moment, without the addition of a single bit of new information." The result: a George Eliot for the New Labour age (I plead guilty), or a queer Queen Victoria for the noughties, or the six wives of Henry VIII refashioned as the post-feminist heroines from Sex and the City.

Why doesn't anyone say or do something about these acts of biographical cannibalism? According to literary agent Andrew Lownie, who runs the Biographers' Club (a networking organisation for practitioners), it's because the power in publishing companies has decisively passed from the commissioning editors to the sales people. "Their thinking is, if something was once a hit, then let's try it again, even though clearly there's a law of diminishing returns. Anything new or different is looked on with suspicion." Lownie's theory is confirmed by the recent experience of one writer I spoke to. Having produced a first biography that punched well above its weight, he has been looking forward to writing his second, this time on a playwright whose work he knows well. His publishers, though, are having none of it and have suggested instead that he try a life of ... Graham Greene. That Norman Sherry's authorised multi-decker Life has sucked every last shred of marrow from Greene's tired old bones seems scarcely to matter.

What makes it doubly strange is that biography as a genre has a tradition of lively experimentation. Consider John Aubrey's Brief Lives, written in the lee of the civil war. If published today, its zany mix of biography and autobiography, gossip and scholarship would be hailed as wonderfully postmodern. Lytton Strachey's Eminent Victorians of 1918 is a fine example of the short, pithy group biographies that are making something of a comeback. These are the fresh, wayward models that I ask my students to consider when they begin their own biographical projects. (Anyone who suggests embarking on a cradle-to-grave narrative of a royal mistress goes to the back of the class.)

In the interests of balance, I should report that it is not only Hamilton who remains bullish about the state of biography. When I asked Claire Tomalin, the biographer of Austen, Pepys and Hardy among others, she could see no reason to worry. Michael Holroyd agrees that "something has definitely changed", though stops short of calling it a crisis. His reading is that popular history and misery memoirs have temporarily knocked biography off its top spot in the non-fiction hierarchy, and that it's probably a question of waiting out the cycle. Michael Prodger, literary editor of the Sunday Telegraph, agrees that the golden age of biography, in which the publication of fat lives of important people by leading writers were automatically given the hushed attention due to a major minor cultural event, has probably passed. But he warns against uncritical nostalgia: "The fact is that there has always been a second division of biography, good enough without being distinguished. These nearly-books are always with us."

The responses of publishers were more upbeat. This could be because they are obliged to sound perky in public, or it may be that being on the receiving end of hundreds of proposals every year gives them a helpfully elevated view. Thus Jenny Uglow, editorial director of Chatto & Windus, herself a distinguished biographer, reports that "interesting ideas are still coming in, although perhaps we're having to look harder for them than we used to". Arabella Pike, meanwhile, who heads up a distinguished biography list at HarperPress, is justly proud of Linda Colley's Elizabeth Marsh: A Woman in World History, which came out last year. An intellectually bold book, Elizabeth Marsh sets itself the task of plotting "the world in the life" of a humble British woman in the 18th century whose own history happened to collide with that of Britain's emerging empire. It is a fascinating book, which speaks straight to the current preoccupation with globalisation, written by one of our leading academic historians.

It has not, however, so far made it on to any of the shortlists for the major literary prizes. The reluctance of prize juries to get to grips with books that do not yield up their charms as easily as a royal mistress in an over-scented boudoir is not something that surprises Holroyd, who believes that scholarly and innovative biographies are increasingly overlooked in favour of texts that, although fluent and timely, are closer to journalism. "One used to award literary prizes to the 'tortoise' book, the one that had taken years of archive work but was never going to be a popular hit," he recalls. These days, by contrast, juries seem keener to make sure that a crowd-pleasing title wins on their watch. Stories slipping out from judging rooms over the past couple of years suggest a growing wariness of books that appear too long, complex, or perhaps simply good-mannered to beg for easy love.

Whenever two or three biographers are gathered together, it isn't long before the names "Amanda Foreman" and "Georgiana" come up. The fact is that much of the devaluing of the biographer's skill can be dated to 10 years ago when Foreman's first book, a biography of the 18th-century Duchess of Devonshire, became a publishing phenomenon. By choosing to be photographed nude behind a pile of books, and by allowing her own life story (starry father, tricky adolescence) to become as important as the person she was writing about, Foreman did an accidental disservice to biography in general, and to young women biographers in particular. Her success with a book written while still in her 20s created unrealistic expectations about the quality of work that young historians are regularly able to produce. Thirty years ago, no one would seriously think that many writers under 30, with or without a PhD, could publish books that would do anything other than slip shyly on to the market, acting as calling cards for the better work that was surely still to come.

But since Foreman's unprecedented hit, photogenic young women are routinely commissioned to produce biographies of equally camera-ready subjects, regardless of whether they are equipped to do so. The results are often intellectually slight and stylistically poor. (Although this isn't always so - in which case, some rather good young women biographers are probably missing out on the critical attention they deserve.) The press releases accompanying these books are always the same, breathlessly referencing what are seen to be the author's key qualifications for the job: she went to Oxford, has done some part-time modelling and has a hobby such as grape-treading or motor-car racing. Whether she knows anything at all about her subject (and, let's be clear, a PhD is not the same as a decade of immersion in the relevant archives) is neither here nor there.

If this seems a peculiar way for publishers to pick their authors, not to mention their subjects, then Lownie can explain. "If a writer is untried, then they are, paradoxically, more attractive than one who already has form. The moment you publish your first book, you leave a paper trail. People can look up your sales figures and make a judgment about whether you are worth giving another chance. A new author, by contrast, is an unknown quantity."

Even older and not especially winsome biographers routinely feel the effect of this "anyone can do anything" culture. I have lost count of the number of times a publishing professional has suggested that I might like to try my hand at a subject from the Tudor or Stuart period (on the unspoken assumption, presumably, that my usual patch, the Victorians, is all used up). If I reply that, while it sounds fascinating, I have absolutely no expertise in the politics or economics (let alone the handwriting, and theology) of the 16th and 17th centuries, I am regarded as a spoilsport and, quite possibly, missing the point.

Still, better to seem like a dreary pedant than a rank amateur. That's what happened six weeks ago to Veronica Buckley when it emerged that, during her writing of a biography about Mme de Maintenon (yet another royal mistress), she had mistaken a fictional diary of Louis XIV for the real thing. For Brotton, Buckley's mistake is the predictable outcome of a systematic deskilling over the past 30 years. "Today's biographers simply aren't immersing themselves in the manuscript sources, but are working instead with more accessible printed versions. As a result, they fail to develop the deep familiarity with their material which would prevent such basic errors."

This reluctance to consult the primary sources is not because biographers have suddenly become lazy or stupid - in general, they are a hunched and harried crew who have to be prized away from their desks - but because they are increasingly given unrealistic deadlines to complete work. It is not uncommon these days for a publisher to allow two years for the writing of a major life. In which case, there is really no time to do anything other than get the relevant books out of the London Library, skim read, and embark on a hectic round of cut and paste (see Raj Persaud, the time-pressed media-friendly psychiatrist who was last week suspended for three months as a result of his over-enthusiastic adoption of this method of "research"). The kind of professional start that Holroyd enjoyed as a young writer in the early 1960s, working for years on the Strachey archive before publishing his first major biography, would be unthinkable now. Chances are that his publishers would lose patience, cancel the contract, and find someone else who could promise to deliver in 18 months.

Yet, despite this gloomy picture of careless publishers lashing on inexperienced writers, there are signs that something important is about to happen in biography. Jonathan Bate of Warwick University, who wrote a prizewinning life of John Clare, declares that "we are on the edge of a paradigm shift". In fact, he believes that he has already seen the future, in the shape of books such as Charles Nicholl's The Lodger: Shakespeare on Silver Street and Frances Wilson's The Ballad of Dorothy Wordsworth, which come at familiar subjects from odd angles. "Instead of old-style literary biography, which tells the story from cradle to grave, these books focus in on just one episode in a life and, in the process, unfold a far richer story." Bate is also justly excited about Holroyd's upcoming A Strange Eventful History, which, rather than concentrating on a single subject, braids together the lives of Ellen Terry, Henry Irving and their families in order to tell a broader story about theatre culture on the eve of the modern age.

Pike is also enthusiastic about the mutation of biography into new, hybrid forms. She points to the work of Robert Macfarlane, who recently blended ecological reportage with auto/biography so memorably in Mountains of the Mind and The Wild Places. She also admires Kate Summerscale's The Suspicions of Mr Whicher, published this year, which cleverly melds family history and crime writing with social and literary history.

To this list I would add, from two years ago, Alexander Masters's Stuart: A Life Backwards - winner of the Guardian first book award - which, like work by Macfarlane and Summerscale, managed to become both a critical and a popular success. The book is about Stuart Shorter, an alcoholic, homeless man whose life story seemed to demand that it be written backwards in order to catch a proper sense of chaos. Masters, a young first-timer now working on his second book, happily admits to being turned off by the idea of writing what most people understand as biography: "It would never cross my mind to do the life of someone dead or great - it would be completely beyond me." (If that sounds admirably modest, Masters also added, "Anyway, who on earth needs another biography of George Eliot - it's so boring!") Instead, he urges biographers to "get out and write about the people under their noses - the people whose stories may seem mundane, but whose daily existences are actually filled with Homeric struggles".

I'm guessing that Stuart was written with a small advance. In order for biographers to make these kinds of breakthroughs in form, it may mean taking less money than they might like. Certainly Uglow urges writers to stick up for the projects they want to pursue, even if they don't initially seem like money-spinners. Her own recent book, Nature's Engraver: A Life of Thomas Bewick, is a masterclass in how these risks can pay off. A biographer with a firmer eye on the market might have passed up the opportunity to write about the artist whose visual renderings of country life went so far towards defining the way the early 19th century saw the natural world. Nonetheless, Bewick ended up as one of the big successes of 2006, mentioned in "book of the year" slots more than any other title.

None of these aforementioned books on its own represents Bate's "paradigm shift", but taken together they do suggest that something significant is astir. Nor should we be complacent that, when the biographical future does arrive, we will necessarily like or even recognise it. The first time I heard about Stuart from Masters's publicist, I thought it sounded glum and dull - a view shared by Masters's then agent, who refused even to take the manuscript out of the Jiffy bag (he was soon replaced). All one can hope, really, is that a small but growing band of writers are financially and intellectually brave enough to strip biography back to its first principle - the recording of lived experience - and consider that brief in the widest possible terms. And even if we don't at first enjoy their efforts, perhaps finding them tricky or strange, at least it means that we will be spared too many more visits from those royal mistresses with their come-hither smiles.