While working in a bookstore in Boone, North Carolina, back in 2011, a 36-year-old college dropout named Hugh Howey started writing a series of sci-fi novellas called Wool. His stories were set in a postapocalyptic world where all human survivors live in an underground silo, a microsociety where resources are so scarce that one person has to die before another can be born. Howey had already published a book with a small press, but he wanted to retain creative control, and he didn't want to go through the arduous process of finding an agent. So he decided to put out the new books himself, selling digital downloads and print editions through Amazon. In the first six months he sold 14,000 copies. Each new installment met with immediate enthusiasm. Within hours he'd receive emails from readers hungry for more.

- Platinum Age of TV

By January of last year, agents were calling Howey, looking to publish the books through more established channels, but he was reluctant. At that point, the Wool series was already making him close to $12,000 a month. Nelson Literary Agency founder Kristin Nelson won Howey over when she admitted that she wasn't sure traditional publishing could offer him anything better than what he was doing on his own. (When she recounted this remark at a recent industry conference, the publishing professionals in the audience shifted uncomfortably in their chairs.) By May, Wool was bringing in $130,000 a month, and Howey and Nelson had sold the film option to 20th Century Fox and Ridley Scott. A couple of publishers made seven-figure offers for the rights to sell the book in hardcover, paperback, and ebook, but Howey and Nelson turned them down. He'd make that much in a year of digital sales alone.

Then Simon & Schuster's president sent Nelson an email that opened the door to a six-figure deal for print rights only. It was an extraordinary concession—the publisher would agree to put its full marketing muscle behind Wool despite having to forgo the ebook revenue stream that has generated the bulk of the series's earnings. It's often said in publishing that with a blockbuster book, everybody wins. But with Wool, it's Hugh Howey who has won biggest.

After centuries in which books and the process of publishing them barely changed, the digital revolution has thrown the entire business up for grabs. It's a transformation that began with the rise of Amazon as an online bookseller and accelerated with the resulting decline of the physical bookstore. But with the shift to ebooks—which now represent upwards of 20 percent of big publishers' revenue, up from 1 percent in 2008—every aspect of the existing framework is now open to debate: how much books will cost, how long they'll be, whether they'll be edited, who will publish them, and whether authors will continue to be paid in advance to write them. It's a future that Amazon doesn't control and one where traditional publishers might eventually thrive, not just survive. The only certainty is that the venerable book business, a settled landscape for so long, is now open territory for anyone to claim.

Of all the worries in the publishing world these days, the king of them is cultural irrelevance. "The fact is that people don't read anymore," Steve Jobs told a reporter in 2008, blurting out the secret fear of bookish people everywhere. But consider this: In one week, people who don't read anymore bought about half a million copies of a really long book called Steve Jobs. In the past year, Vintage has sold one book from the Fifty Shades of Grey trilogy for every six American adults. The Big Six publishers—Random House, Penguin, Hachette, Macmillan, Simon & Schuster, and HarperCollins—all make money, and at profit margins that are likely better than they were 50 years ago.

Meanwhile, readers have an unprecedented array of options. E-readers have gotten consistently cheaper and better since the first Kindle shipped in 2007, giving customers instant access to millions of titles. For a couple of dollars you can buy a self-published sensation or a Kindle Single rather than a full-length book. Add it all together and you have a more vibrant market for literary material than ever before, with nearly 3 billion copies sold every year. Amazon likes to point out that new Kindle buyers go on to purchase almost five times as many books from Amazon, print and digital, in the ensuing year as they did in the prior one. "I believe we'll look back in five years," says Russ Grandinetti, VP of Kindle content for Amazon, "and realize that digital was one of the great expansions of the publishing business."

For all the digital optimism, not even Amazon is ready to declare the traditional model dead. In May 2011 the company announced that it was going head-to-head with the Big Six by launching a general-interest imprint in Manhattan, headed by respected industry veteran Larry Kirshbaum. It signed up celebrity authors, paying a reported $850,000 for a memoir by Laverne & Shirley star Penny Marshall and winning over best-selling self-help author Timothy Ferriss. Tired of being undersold by Amazon and wary of its encroachment into their business, many brick-and-mortar booksellers refused to stock the titles. The boycott has worked so far: Marshall's book flopped, and Ferriss' undersold his previous offering. Ferriss says he doesn't regret his experiment with Amazon Publishing, but he allows, "I could have made more money—certainly up to this point—by staying with Random House."

Still, it's not clear that traditional publishers are well positioned to own the digital future. They are saddled with the costs of getting dead trees to customers—paper, printing, binding, warehousing, and shipping—and they cannot simply jettison those costs, because that system accounts for roughly 80 percent of their business. Ebooks continue to gain ground, but the healthiness of the profit margins is unclear. J. K. Rowling's latest book helps illustrate this bind. At a rumored advance of $7 million, Little, Brown essentially backed up an armored car to Rowling's house to pay her before seeing a nickel in revenue. The publisher then paid highly trained people to improve the novel and well-connected people to publicize and market it until it was inescapable. Little, Brown's landlord in Manhattan occasionally asks for rent too. If a reader can buy the Kindle edition for $8.99, the public might eventually find it absurd to pay $19.99 for a printed version, let alone the $35 that Little, Brown wants for the hardcover.

What's more, awarding huge contracts for books that may not even be written yet creates tremendous risk. The industry is plagued by what indie-publishing entrepreneur Richard Nash has called the "pathology of unearned advances." An author who gets a book deal is paid an advance against royalties, and if the royalties end up exceeding the advance, the author starts getting more checks. But that doesn't usually happen.

The uncertainty about a book's potential value cuts both ways. Daniel Menaker, former executive editor in chief of Random House, told me what happened when a fellow editor there presented a case to his colleagues for making an offer on Laura Hillenbrand's Seabiscuit: "People just laughed, and someone said, 'Talk about beating a dead horse.' " Good one! The editor, Jonathan Karp, luckily won the argument, and Random House bought the rights for only five figures. More than 2 million copies were in print even before the movie came out. Unfortunately, the more common scenario is that a publisher opens the vault for a book that tanks. Bantam paid a reported $2 million in 2005 for two novels from a sci-fi writer named Gordon Dahlquist. If the title The Glass Books of the Dream Eaters doesn't sound familiar, you're not alone.

>The disappearance of the physical bookstore would endanger the entire book business—even Amazon.

The publishing houses stay afloat only because the megahits pay for the flops, and there's generally enough left over for profit. Predicting the success or failure of any given book is impossible. Menaker recalls Jason Epstein, who led Random House for four decades, telling him, "Make no mistake—this is gambling." Which is why the pricing pressure on ebooks is so scary to publishers: If they are the gambler at the slot machines, placing scores of bets and relying on the winnings to trump the losses, Amazon represents a casino that offers smaller and smaller payouts.

Beyond the immediate concern over prices, publishers also worry that the disappearance of the physical bookstore could endanger the entire book business, even (ironically) Amazon. Research has shown that readers don't tend to use online bookstores to discover books; they use them to purchase titles they find out about elsewhere—frequently at physical stores. (If you want to see a bookstore owner get angry, mention Amazon's Price Check app, which allows customers to scan an item in a physical store and buy it for less from Amazon then and there.) With no stores to browse in, publishers fear, book sales everywhere could take a significant hit.

This is one reason that, in 2010, five of the Big Six publishers worked with Apple to institute a new model to keep other retailers competitive with Amazon. In an attempt to win customers, Amazon had been routinely selling ebooks at a loss, paying, say, $12 to $15 wholesale for a popular ebook and then selling it for $9.99. Under the new so-called agency model, the publisher would have the power to set the price everywhere—between $12.99 and $14.99 for most best sellers—but the retailer would take 30 percent. That is, the publishers agreed to a scheme in which Amazon would make significantly more per book and they would make less. They were playing the long game, trying to protect physical stores and print sales and chip away at Amazon's overwhelming ebook market share.

After fighting the plan, Amazon caved. But the Department of Justice sued the five publishers and Apple for collusion, and Amazon described one of the resulting settlements as "a big win for Kindle owners." The recently announced merger of the two biggest of the Big Six, Random House and Penguin, is widely seen as a move to build an entity that can stand up to Amazon's market power.

In the long term, what publishers have to fear the most may not be Amazon but an idea it has helped engender—that the only truly necessary players in the game are the author and the reader. "I was at a meeting God knows how many years ago at MIT," former Random House chief Epstein says, "and someone used the word disintermediation. When I deconstructed that, I said, 'Oh my God, that's the end of the publishing business.' " At a time when a writer can post a novel online and watch the revenue pour in by direct deposit, the publishing industry's skill at making books, selling them by hand to bookstores, and managing the distribution of the product threatens to become irrelevant. In Epstein's vision, the writer may need a freelance editor, a publicist, and an agent who functions as a kind of business manager, but authors will keep a bigger share of the proceeds with no lumbering media corporation standing in the way.

So far this phenomenon has largely been limited to previously unknown writers like Hugh Howey. Amanda Hocking, a 26-year-old Minnesotan who worked days at an assisted-living facility, grossed about $2 million on ebooks in a little over a year with her paranormal romances and zombie novels for young adults. John Locke, a self-published crime writer, had already beaten Hocking to the 1-million-ebook mark on Amazon. And then, of course, there is E. L. James' Fifty Shades of Grey, which began as self-published Twilight fan fiction but wound up making 2012 so bountiful for Random House that it gave a $5,000 bonus to each employee.

But these are the exceptions. In general, new writers gain much more than they lose by signing with a major house. Most self-published authors have trouble selling a copy outside of their immediate family. Even if they have talent, they lack professional help or the imprimatur of quality that a publisher can bring. Indeed, Fifty Shades, which some have taken to be the definitive evidence in favor of self-publishing, is more accurately a demonstration of the opposite: The book became a massive commercial success only after Random House got involved, placing giant stacks of paperbacks in bookstores everywhere and buying huge ads in the London Underground.

The real danger to publishers is that big-ticket authors, who relied on the old system to build their careers, will abandon them now that they have established an audience. As Howey says, "When that happens, all bets are off." The John Grishams of the world already manage to extract excellent deals in the traditional way because of their huge and reliable sales, and few writers relish the work of being their own publisher. But as that work grows easier—as complex print distribution loses ground to low-cost digital delivery—the big names are starting to get tempted. Stephen King has been experimenting with bypassing his publisher, releasing his latest essay as a Kindle Single directly (albeit with some editing and promotion) through the Amazon store. The popular suspense writer Barry Eisler turned down a $500,000 book contract with the intention to self-publish—but before he could do so, Amazon Publishing offered him a sweetheart deal.

Pretty soon one of these famous writers will step up to the cliff and actually jump. Maybe it will be Tim Ferriss. His less-than-stellar results with Amazon might push him back to a traditional publisher—or in another direction entirely. A great deal of money hinges on what he and his fellow best-selling authors decide to do next. "I wouldn't be surprised if I self-published in the next few years," Ferriss says. "Wouldn't remotely surprise me."

Evan Hughes (@evanhughes) is the author of Literary Brooklyn.