Earlier this week, Audible revealed that it was working on a new feature for its audiobook app: Audible Captions, which will use machine learning to transcribe an audio recording for listeners, allowing them to read along with the narrator. While the Amazon-owned company claims it is designed as an educational feature, a number of publishers are demanding that their books be excluded, saying these captions are “unauthorized and brazen infringements of the rights of authors and publishers.”

On its face, the idea seems useful, much in the same way that I turn on subtitles for things that I’m watching on TV, but publishers have some reason to be concerned: it’s possible that fewer people will buy distinct e-book or physical books if they can simply pick up an Audible audiobook and get the text for free, too.

And Audible may not have the right to provide that text, anyhow.

In the publishing world, authors and their agents sign very specific contracts with publishers for their works: these contracts cover everything from when the manuscript needs to be delivered, how an author is paid, and what rights to the text a publisher might have, such as print or audio. As an audiobook publisher and retailer, Audible gets the rights to produce an audiobook based on a book, or to sell an audiobook that a publisher creates in its store. Publishers say that a feature that displays the text of what’s being read — itself a reproduction from the original text — isn’t one of those specific rights that publishers and authors have granted, and they don’t want their books included in Audible’s feature when it rolls out.

But Audible Captions aren’t quite the same experience as reading an e-book, as you can see here:

Audible tells The Verge that the captions are “small amounts of machine-generated text are displayed progressively a few lines at a time while audio is playing, and listeners cannot read at their own pace or flip through pages as in a print book or eBook.” Audible wouldn’t say which books would get the feature, only that “titles that can be transcribed at a sufficiently high confidence rate” will be included. It’s planning to release the feature in early September “to roll out with the 2019 school year.”

Penguin Random House, one of the world’s five biggest publishers, told The Verge that “we have reached out to Audible to express our strong copyright concerns with their recently announced Captions program, which is not authorized by our business terms,” and that it expects the company to exclude its titles from the captions feature.

‘an unauthorized and brazen infringements of the rights of authors and publishers’

Other publishers have followed suit. Simon & Schuster (disclosure: I’m writing a book for one of its imprints, Saga Press), echos their sentiments, calling the feature “an unauthorized and brazen infringements of the rights of authors and publishers, and a clear violation of our terms of sale,” and has also told Audible to “not include in Captions any titles for which Simon & Schuster holds audio or text rights.” A Macmillan spokesperson said that “the initiative was not authorized by Macmillan, and we are currently looking into it.”

The Authors Guild also released a statement, saying that “existing ACX and Audible agreements do not grant Audible the right to create text versions of audio books,” and that the feature “appears to be outright, willful copyright infringement, and it will inevitably lead to fewer ebook sales and lower royalties for authors for both their traditionally published and self-published books.”

When asked about the feature squares up against the existing audio rights that are granted to it, an Audible spokesperson told The Verge that it does “not agree with this interpretation,” but declined to comment further on whether or not the company actually has the right to go through with it. Audible refused to comment about whether or not it would honor publishers’ requests, saying only that it was working with publishers to “help address some confusion about how Audible Captions works and what listeners will experience.”

The Verge has reached out to other major publishers for comment, including Little, Brown and Company and Hachette, but did not hear back by publication of this post.

DongWon Song, an agent from the Howard Morhaim Literary Agency, tells The Verge that there’s two sets of issues at play, because Audible acts as both a publisher and as a retailer. “If Audible is producing a book as a sublicense from a publisher,” he says “then they only have rights to the audio, or they have the right to create and sell an audio recording of the text, and that’s all the rights that they’re given: they’re not given any text rights, they’re not given any electronic rights, they’re just given the right to distribute an audio edition.”

“As a retailer, they have even less rights. As a retailer, they have no right to what’s in the text, so if it’s third party, whether Macmillan makes the audio or Random House or whatever, they make their own audio edition, and as a technology or sales platform, Audible only has the right to sell those things on the platform.”

Audible’s feature is using machine learning to translate those audio recordings into text, and while the company denied that it’s creating ebooks, that translation is “not a thing that is explicitly granted to them,” Song says. “I hear what they’re saying on one level, but on another level, they are reproducing the full text of the book in print form. Full stop. It’s not an excerpt, it’s not a fair use argument, they’re taking the entire text and reproducing that on your phone or on your device.”

‘they are reproducing the full text of the book in print form. Full stop.’

Song notes that while he sees some value in such a feature, “it’s depriving authors of a route for more income and to be paid for their work.”

This isn’t the first time that Amazon has come under fire for publishers when it comes to translating text to audio, or vice-versa. In 2009, the company backtracked on a text-to-speech feature on the Kindle, which allowed readers to listen to their book with machine-generated narrator. The Authors Guild argued that the feature deprived authors of their audio rights, and Amazon disabled it.