Photo: Paul Ruscha/© Ed Ruscha/Courtesy of Ed Ruscha and Gagosian Gallery (“Me”, 2001)

How-to writers are to other writers as frogs are to mammals,” wrote the critic Dwight MacDonald in a 1954 survey of “Howtoism.” “Their books are not born, they are spawned.”

MacDonald began his story by citing a list of 3,500 instructional books. Today, there are at least 45,000 specimens in print of the optimize-everything cult we now call “self-help,” but few of them look anything like those classic step-by-step “howtos,” which MacDonald and his Establishment brethren handled only with bemused disdain. These days, self-help is unembarrassed, out of the bedside drawer and up on the coffee table, wholly transformed from a disreputable publishing category to a category killer, having remade most of nonfiction in its own inspirational image along the way.

Many of the books on Amazon’s current list of “Best Sellers in Self-Help” would have been unrecognizable to MacDonald: Times business reporter Charles Duhigg’s The Power of Habit, a tour of the latest behavioral science; Paulo Coelho’s novel The Alchemist, a fable about an Andalusian shepherd seeking treasure in Egypt; Susan Cain’s Quiet: The Power of Introverts in a World That Can’t Stop Talking, a journalistic paean to reticence; publisher Will Schwalbe’s memoir The End of Your Life Book Club, about reading with his dying mother; and A Child Called “It,” David Pelzer’s recollections of harrowing and vicious child abuse. And these are just the books publishers identify as self-help; other hits are simply labeled “business” or “psychology” or “religion.” “There isn’t even a category officially called ‘self-help,’ ” says William Shinker, publisher of Gotham Books. Shinker discovered Men Are From Mars, Women Are From Venus and now publishes books on “willpower” and “vulnerability”—“self-help masquerading as ‘big-idea’ books.”

Twenty years ago, when Chicken Soup for the Soul was published, everyone knew where to find it and what it was for. Whatever you thought of self-help—godsend, guilty pleasure, snake oil—the genre was safely contained on one eclectic bookstore shelf. Today, every section of the store (or web page) overflows with instructions, anecdotes, and homilies. History books teach us how to lead, neuroscience how to use our amygdalas, and memoirs how to eat, pray, and love. The former CEO of CNN writes the biography of an ornery tech visionary and it becomes a best seller on the strength of its leadership lessons. The Nobel-laureate psychologist Daniel Kahneman writes a subtle analysis of our decision-making process and soon finds his best seller digested and summarized in M.B.A. seminars across the country. Philosophical essayist Alain de Botton launches a series of self-help books called “The School of Life,” whose titles will all begin with “how to.” Even before books are written, their advances are often predicated on strong “takeaways” targeted to proven demographics. More like a virus than MacDonald’s frogs, self-help has infiltrated and commandeered other fields in its drive to reproduce. This plague of usefulness has burrowed its way into the types of books that were traditionally meant to enlighten, or entertain, or influence policy, but not exactly to build better selves. It’s generally led to better self-help, more grounded in the facts and narratives that drive the other genres, but also to a nonfiction landscape in which every goal is subjugated to the self-­improvement imperative.

This new kind of self-help could never thrive in a vacuum. Or rather, it thrives in a particular vacuum—the one left behind by the disappearance of certain public values that once fulfilled our lives. Strains of self-help culture—entrepreneurship, pragmatism, fierce self-­reliance, gauzy spirituality—have been embedded in the national DNA since Poor Richard’s Almanack. But in the past there was always a countervailing force, an American stew of shame and pride and citizenship that kept these impulses walled off, sublimating private anxiety to the demands of an optimistic meritocracy. That force has gradually been weakened by the erosion of all sorts of structures, from the corporate career track to the extended family and the social safety net. Instead of regulation, we have that new buzzword, self-regulation; instead of an ambivalence over “selling out,” we have the millennial drive to “monetize”; and instead of seeking to build better institutions, we mine them in order to build better selves. Universities now devote faculty to fields (positive psychology, motivation science) that function as research arms of the self-help industry, while journalists schooled in a sense of public mission turn their skills to fulfilling our emotional needs. But since self-help trails with it that old shameful stigma, the smartest writers and publishers shun the obvious terminology. And the savviest readers enjoy the masquerade, knowing full well what’s behind the costume: self-help with none of the baggage.

It was in the seventies that we began to shed that baggage, starting with the outer layer of self-help: common sense. Children of the postwar middle class were weaned on the mass paperbacks of Dr. Spock, and their parents learned how to win friends and think positively from Dale Carnegie and Norman Vincent Peale. But in the late sixties, that gray-flannel-suit howtoism gave way to the reemergence of an older, more mystical strain, part bootstrapping and part magical thinking. The New Age was really a revival of what had once been called New Thought: a religious movement spawned in the primordial soup of Ralph Waldo Emerson, Sigmund Freud, and William James that preached the flip side of the Protestant work ethic: faith above works and a belief in one’s unlimited capacities on Earth. The new New Thought was the perfect religion for the Me Decade, a ­reality-show version of spirituality in which the meaning of life is to unleash the inner superstar.

You might date the final triumph of New Thought over mid-century pragmatism to the relocation of Harper & Row’s venerable religious division. In 1977, the old Protestant imprint moved to New Age–soaked San Francisco, land of Esalen, yoga, est, and Human Potential. Nine years later, it partnered with the Hazelden clinic to publish Melody Beattie’s Codependent No More. Suddenly, the jargon of AA became the jargon of the USA. Linda Loewenthal, who led self-help beacon Harmony Books, calls the recovery boom “my awakening to the power of naming something.” And, actually, “recovery” named everything, defining every problem as a personal illness to be conquered—toxic parents, women who love too much, obesity, excessive shopping, and above all “codependency,” which could potentially encompass any human relationship.

Recovery-inspired self-help replaced doctors, priests, and therapists (and maybe even parents, senators, and teachers) with public personalities who gave names to the problems of millions. In the insecure nineties, these Martin Luthers translated elite (and expensive) knowledge into news Americans could use. Suze Orman had worked at Merrill Lynch before ending up a financial counselor to the recently laid off. Then she pitched a book to Esther Margolis, the head of self-help publisher Newmarket Press. Now Orman’s the preeminent adviser to a downsized middle class. Deepak Chopra was a doctor at Tufts and Boston University who turned to meditation. He went to Harmony Books with his 1993 breakthrough, Ageless Body, Timeless Mind: The Quantum Alternative to Growing Old. In one soothing voice, East met West, the mind met the body, and the aging boomers met their age-defying guru.

The more the healers (and their “conditions”) proliferated, the harder it became for customers to figure out where to focus their limited time, money, and attention. It made sense that by 2000, our biggest guru wasn’t a writer but a new pope, Oprah Winfrey, whose brand lay in the power to ordain others. Like that other millennial guide, ­Malcolm Gladwell, she was a curator rather than a creator. But every one of her favorite things reflected her glow, from the corniest health program to the strangest New Age manifesto. Eckhart Tolle’s metaphysical The Power of Now had an initial printing of 3,000 copies in 1997. Five years later, around the time Oprah named it a personal favorite, it became a best seller. So did equally esoteric The Secret, a magical-thinking throwback to New Thought. Even the novels she selected, from The Corrections to The Road, instantly became not just blockbusters but launching points for discussions about relationships and personal growth—howtos with an aesthetic kick.

Oprah helped create a landscape in which anything can be self-help—so long as it isn’t the narrowly defined howtoism of old. Greg Brandenburgh, an alumnus of Harper San Francisco, once made a staple of illness primers. “What you’re looking for is to publish on conditions that are chronic and incurable,” he half-jokes. But where recovery books pathologized the public, current best sellers target what Heather Jackson, an editor at Harmony, calls “the worried well looking to optimize, to make their lives that much better.” Jackson saw this coming when she took on Timothy Ferriss’s The 4-Hour Workweek. His life-­hacking manifesto aims to cure not a sickness but something we used to consider a fairly natural state—inefficiency.

The self-made Ferriss notwithstanding, the peddlers of the new efficiency don’t much resemble the gurus of old. More ­often, they present themselves humbly as journalists or social scientists exploring new research frontiers for our benefit. “Somebody’s gonna kill me for saying this,” says the editor Caroline Sutton, when asked what kicked off the trend of essayistic self-help. “But I think it was The Tipping Point,” Gladwell’s 2000 argument for the power of social connections, which made it safe for techies and business types—and, more generally, men—to read about bettering themselves. “The whole idea of showing that there is a counter­intuitive way of looking at information, to make you understand yourself in a completely different way—that’s been game-changing,” Sutton says.

You could argue that the marriage of self-help and social science began a few years earlier with Daniel Goleman, a bridge between self-help’s New Age past and its journalist-driven, label-defying present. A Harvard Ph.D. and a science reporter for the New York Times, Goleman had also co-written a book on meditation with psychedelic pioneer Ram Dass. For a book subverting the importance of IQ, he went to editor Toni Burbank, a veteran self-help guru at Bantam Books. ­Emotional Intelligence: Why It Can Matter More Than IQ spent more than a year on the Times best-seller list—its general rundown, not the ghetto of “Advice, How-To, and Miscellaneous.”

It was Goleman who introduced Loewenthal, now an agent, to the science journalist Sharon Begley. Her book, which she wanted to title Train the Mind, Change the Brain, touted fMRI scans showing that meditating Buddhist monks could literally reshape their brains. Loewenthal sold the book to Sutton, then at Random House and now arguably the reigning queen of “Big Idea” self-help. Sutton publishes the Penguin imprint Hudson Street Press, putting out at least a dozen titles per year explicitly focused on “nonfiction with a strong takeaway.” Sutton eventually suggested changing Begley’s title to Train Your Mind, Change Your Brain, and with those two pronouns, a 2007 self-help best seller was born.

“I think it’s important to offer out a promise to the reader,” Sutton says. “It’s a kind of quintessentially American thing.” What distinguishes crossovers like Begley is the level of scientific detail behind the promise. The guru has given way to the data set—as explicated by journalists eager to break the constraints of a shrinking medium by pitching their discoveries directly to the masses. And where yesterday’s healers had their Esalen Institute and ­Hazelden, journalists and scientist-writers have lecture circuits. “TED,” says Sutton, “is sort of like the new Oprah.”

“An increasing segment of the market wants to read about the synthesis of different modalities,” says Loewenthal. What she means to say, probably, is that we are in a new era of mass self-help, wherein the laboratory and the writer work together to teach us how to change ourselves, rather than our world. Take the latest subgenre, Positive Psychology. Like so many movements of the past, it began as the province of professionals pursuing greater knowledge or maybe better policy—until it was brought down to Earth by an enterprising hand-holder. Her name is Gretchen Rubin, and she’s also on Amazon’s self-help best-seller list, with her 2009 book, The Happiness Project: Or, Why I Spent a Year Trying to Sing in the Morning, Clean My Closets, Fight Right, Read Aristotle, and Generally Have More Fun. It’s full of data, but chummily narrated within the framework of a memoir. It bears all the hallmarks of contemporary self-help, including its shyness about the label. (Rubin’s first book was actually a parody of self-help.) But for all its modern pretensions, it focuses, like all the enduring best sellers, on a chronic and incurable ­American flaw: unhappiness.