Anyone wanting to damage their intellectual credentials at a stroke need only do one simple thing: confess they read self-help books.

There's no more ridiculed genre in the literary canon – and you can see why. Most self-help books are written by Americans of the most sentimental and unctuous sort. They promise their readers eternal life, untold riches and an escape from every grubby aspect of being human, all within 300 pages of upbeat, relentlessly repetitive and patronising prose. No wonder the unstated assumption of the cultural elite is that really only stupid people read them.

What about everybody else? The assumption is that life doesn't need to be navigated with lessons. You can just do it intuitively. After all, you only need to achieve autonomy from your parents, find a moderately satisfying job, form a relationship, perhaps raise some children, watch the onset of mortality in your parents' generation and eventually in your own, until one day a fatal illness starts gnawing at your innards and you calmly go to the grave, shut the coffin and are done with the self-evident business of life.

However, most of us will probably privately admit that living isn't entirely as simple as that – and that it might be useful to have somewhere to turn. For 2,000 years in the history of the west, most of philosophy simply was self-help. The Ancients were the most adept practitioners. Epicurus wrote some 300 self-help books on almost every topic, including On Love, On Justice and On Human Life. The Stoic philosopher Seneca wrote volumes advising his fellow Romans how to cope with anger (the still very readable On Anger). It is no injustice to describe Marcus Aurelius's Meditations as one of the finest works of self-help ever written, as relevant to someone facing a financial meltdown as the disintegration of an empire.

Christianity continued in this vein, with such bestselling guides as Thomas a Kempis's The Imitation of Christ. What, then, explains the gradual decline in the prestige of self-help books that continues to this day? A key catalyst was the development of the modern university system that in the mid-19th century became the main employer for philosophers and intellectuals and started to reward them not for being useful or consoling, but for getting their facts right. There began an obsession with accuracy and a corresponding neglect of utility. The idea of turning to a philosopher or historian in order to become wise (an entirely natural assumption for our ancestors) started to seem laughably idealistic and adolescent. Alongside this came a growing secularisation of society, which emphasised that the modern human being could do the business of living and dying by relying on sheer common sense, a good accountant, a sympathetic doctor and hearty doses of faith in science. The citizens of the future weren't supposed to need lectures in how to stay calm or free of anxiety.

And so the self-help field was abandoned to the many curious types who thrive in it today: people who are reclothing the Christian message so as to promise us financial heaven if we believe in ourselves, have faith, work hard and don't despair. Or else those with a passing acquaintance with Buddhism, psychoanalysis or Daoism. What unites modern practitioners is their fierce optimism. They make the grave assumption that the best way to cheer someone up is to tell them that all will be well. They are utterly cut off from the spirit of their more noble predecessors, who knew that the fastest way to make someone feel well is to tell her that things are as bad as, and possibly much worse than, she could ever have thought. Or, as Seneca put it so well, "What need is there to weep over parts of life? The whole of it calls for tears."

To try to rectify the situation a little, I've just edited a new series of six self-help books with solid intellectual ambitions, authored by experts in their fields, including work, emotional health, technology, money and political activism. The tone is helpful but realistic. There's no earnestness or patronisation. I've written a little volume on sex, with advice on impotence, oral sex, sex addiction, infidelity and other issues.

A culture which gives a role to guidance and the self-help book stands a chance of making at least one or two fewer mistakes than the previous generation in the time that remains.