It’s been an awkward few days for America’s most powerful books ranking. On Aug. 24, the New York Times issued a rare correction to one of its bestseller lists, after a strange and still unexplained series of events that fall somewhere between scam and gaffe.

Last week, authors and booksellers in the young adult books community began tweeting about paranormal romance novel Handbook for Mortals by debut author Lani Sarem, which had appeared with little fanfare at the top of the Times’ list for bestselling hardcover books for young adults. Sarem, people on social media claimed, had gamed the Times’ ranking by buying her own book. Publisher GeekNation, the entertainment site that launched a book imprint with this title, has yet to respond to a request for comment from Quartz.

Social media justice was swift and brutal: Sarem’s book was dropped from the New York Times’ list, and now she’s facing an arguably worse problem. On Amazon, people are giving her one-star reviews, citing the rumors about her alleged gaming of the list: “Try harder if you want to pay your way to the NYT list,” says one. Another says, “If you need to buy your way on to the NYT, you don’t even deserve a one-star review.” One simply says, “You know why.”

The Las Vegas, Nevada-based actor and band manager describes her debut novel as urban fantasy and a paranormal romance. Sarem first wrote Handbook for Mortals, about a woman who joins a magic show, as a script and was urged by a publisher to turn it into a book series, ahead of making a movie. She plans to self-produce a film with actor Thomas Ian Nicholas, who played Kevin in the American Pie teen sex movies.

It’s difficult to say whether Sarem gamed the bestseller ranking system. In interviews with Quartz, she portrays herself as a bullied newcomer and insists she behaved as any other author would. But this skirmish over the Times’ list reveals how little we know about book sales; how flawed the book rankings are; how opaque the Times’ list is; how a book without genuine national interest can top one of the most widely known bestseller lists; and how, with enough insider information, the entire system could indeed be gamed.

How the game began

Phil Stamper, a New York-based author of YA fiction, set things off on Aug. 24 with a tweet expressing skepticism that an unknown book out of stock on Amazon deserved its place as the Times’ bestselling YA book. Booksellers started contacting Stamper, he says, reporting strangely placed orders for Handbook for Mortals that didn’t seem to stem from genuine interest. The amateur sleuthing reached the New York Times by the end of the day, which, following its own investigation, removed Sarem’s book from its list.

“Prompted by ​expressions of concern about the ranking of ’Handbook for Mortals,’ editors had to investigate the issue after the initial ranking was published,” the Times said in an email to Quartz. “They then decided that the sales simply did not meet our criteria for inclusion.” The newspaper provided no further details.

The Times’ list is not a complete ranking of all US books and their sales; rather, the paper compiles its list manually based on what certain bookstores report across the country. The stores, which are kept confidential, use an online portal to report sales numbers on individual books directly to the Times. The Times does not say how it “statistically weighs” sales to “represent and accurately reflect all outlets proportionally nationwide.” Exactly how many book sales are required to make it on the list can vary from week to week.

People try to buy their own books to make it look like they are bestsellers all the time and even pay huge sums to third-party firms to do the work for them. The Times knows this and is usually good at catching self-orders. If the Times knows that a book’s sales include bulk orders, but decides to list the book in the ranking anyway, it adds a †, to show readers they should be wary of the bestseller status. Sarem’s book was ranked number one in YA without a †, which suggested that fans bought up copies individually.

The Times’ list is prestigious, but it isn’t the authority on book sales. The industry go-to, NPD BookScan, represents 85% of print sales in the US and says Handbook for Mortals sold 18,000 print copies in its first week, starting August 15. That is an astounding figure for a little-known book: In comparison, Colson Whitehead’s hotly anticipated The Underground Railroad sold 17,500 print copies during its first week in August last year. Whitehead was then a Pulitzer Prize-nominee and had gotten the holy grail of endorsements, a stamp of approval from American talk show host Oprah Winfrey.

NPD Book counts pre-orders and bulk orders toward a book’s sales. The group says in an email that the 18,000 figure for this book didn’t show an “organic sales pattern” and that ”a substantial volume of POS [point-of-sales] reporting were attributed to bulk sales,” and so it left Handbook off its own bestseller list that week. NPD declined to say exactly how many of the sales were bulk.

Where the book orders came from

Who ordered all those books? Reports from booksellers who received orders for Handbook for Mortals suggest at least some buyers weren’t ordinary fans. Cheryl Peevyhouse, a YA author and an employee at Hicklebee’s, an independent children’s bookstore in San Jose, California, tells Quartz that she answered a call on Aug. 19 from a buyer who first asked if the store reports its sales to the Times—it does—and then placed an order for nearly 90 books. The caller said he was making a movie based on the book and wanted to place an order for an event. The caller didn’t specify the name of the event, or when it was.

Peevyhouse did not have the book in stock, and checked its distributor, Ingram, for copies. Ingram also did not have copies in stock. The caller ordering the 90 books said he knew no books were available yet, but that the publisher had assured him they were on the way. The caller insisted that the order be placed that day, says Peevyhouse, even though she warned that she would not be able to guarantee a delivery date in time for an event. She placed an order with Ingram later in the day. (Ingram confirms to Quartz that there was a lag time between when sales for Handbook started and when the company received copies from the publisher, but declined to say how long it was.)

Peevyhouse wasn’t the only bookstore employee to get such an order request. An employee at a Las Vegas-based Barnes & Noble, who asked not to be named, told Quartz that the store received two phone orders placed one week apart, each for 29 books. According to the employee, 30 books is the number at which an order is marked as a corporate order for Barnes & Noble, which could be excluded when the store reports to the Times. Peevyhouse said Hicklebee’s has no such threshold, nor does it list how many individuals place orders for each book when it reports to the Times. So it’s up to the Times editors’ discretion to decide whether they see “29” and decide it’s interest from 29 or fewer individuals, or 29 books ordered by one person.

In theory if a person orders a book and it never ships, he or she can cancel the order and get a refund. That means it would be possible for someone to create the appearance of high demand in a short period—say, just enough to qualify for the bestseller list that week—by placing big orders, then canceling them later.

According to the Hollywood Reporter, Nicholas, Sarem’s co-promotor, said he did indeed contact indie bookstores—which Sarem says included Hicklebee’s—to buy bulk copies ahead of fall appearances, knowing print copies wouldn’t be available until Aug. 31. “Maybe that’s where things got convoluted,” he told the magazine. Nicholas has been promoting the book at Wizard World comic cons this year in Philadelphia, Pennsylvania; Sacramento, California; Columbus, Ohio; and this past weekend, in Chicago, Illinois.

A wrinkle in the plot

Did Sarem and Nicholas order 18,000 books themselves in tiny batches, to obscure the fact that these were bulk sales? Sarem told Quartz that she and Nicholas bought about 5,000 copies (with a list price of $25 each) for their events and that fans pre-ordered approximately 10,000 at events she did in the first half of the year. The 5,000 the duo bought were for their appearances in Chicago, which is an astronomically high number for first-time authors to try to sell at events.

“That’s an order of magnitude too high” for an author event, says David Joslin, manager of Mysterious Galaxy Bookstore in San Diego, California, which sells books at the 130,000-person Comic Con International. “Even with GRRM, which is the biggest name I can think of that we’ve dealt with, we wouldn’t be selling that high,” he says, referring to George R. R. Martin, author of the books on which the TV show Game of Thrones is based.

NPD seems to contradict Sarem’s statement, saying that based on the timing of the sales data, the group doesn’t have any evidence that a sizable number of the 18,000 were individuals trying to order the book ahead of its release.

Sarem maintains that she behaved as any author would, ordering books at locations where she had scheduled events. She denies any orchestrated attempt to trick the Times. She denies, for instance, using a third-party professional agency that promises bestseller status. Sarem says she and Nicholas have attended ten events this year and are slated for another 12. Perhaps this is a simple case of extreme overconfidence, and the two really do anticipate insanely large crowds and have accordingly spent $125,000 on books that seem likely to go unsold.

The New York Times hasn’t said specifically if Handbook for Mortals gamed its system, but it released an updated list the evening of Aug. 24 and provided a statement saying the book’s sales “did not meet [their] criteria for inclusion.” Sarem says, “I think they just caved to social media pressure. It was too much for them to deal with, and who was I?”

Real buzz required

Until the list kerfuffle, Sarem was indeed a nobody by mainstream publishing standards. The bookstores in cities where you’d expect to see a massive bestseller stacked face-out in storefronts and alongside sold-out author events, have not heard of the book. In New York, Quartz spoke to the Strand, Greenlight, and Books of Wonder, which report their sales to the Times, and to the City Lights Bookstore in San Francisco. None have any copies in the store or have sold any.

Chicago’s Unabridged and Women & Children First bookstores both report to the Times and said they don’t carry the book, haven’t ordered it, and haven’t gotten any requests for it. Mysterious Galaxy Bookstore, which reports to the Times, similarly had no activity for the book, and didn’t receive any pre-order requests at the massive Comic Con this year. Writer’s Block, an independent bookstore in Las Vegas which also reports to the Times, said they hadn’t had any customers come to the store asking for the book, either.

Handbook for Mortals has received little attention on Amazon, where one would expect online pre-orders. There, the book is ranked number 2,457 in teen literature sales at time of writing.

It is of course possible to buck the normal ways of selling and self-promotion in publishing, and sell outside the mainstream publishing institutions: Self-published romance writers have flourished without any gatekeepers by cultivating their own readership and communities online. Publisher Simon & Schuster dropped Milo Yiannopoulos, yet the controversial conservative sold 18,000 books his first week, thanks to his massive online following.

But an author still needs to have genuine buzz, wherever it is, and that doesn’t seem to exist for Handbook for Mortals.

If Sarem is right about the 10,000 genuine fan orders, it’s still not clear why Ingram, the world’s largest wholesale distributor, was left without copies if Sarem’s publisher expected demand to be so high. (Sarem did not provide any information on how many books have been, or were initially, printed.) And we still don’t know why the Times removed her book, instead of editing the listing with a bulk-order †. Was it a mistake to take Handbook off the list, or a mistake to include it in the first place?

Because the details of what counts for the New York Times list is based on certain bookstores, the veracity of their reporting systems, and the discretion of editors, there are a lot of places where the secretive count can go wrong. The way people treat the list is also problematic: As I’ve written before, both the book that spends one week at the bottom of a niche category and the book that spends 35 weeks hovering between numbers 1 and 3 in general fiction can get a “New York Times bestseller” sticker on the cover, and both authors can claim the same accolade on their websites or use it to book speaking gigs.

Ultimately, it’s not that bestselling booklists are meaningless; they just aren’t always meaningful.