Now the author and publisher groups have to decide whether to resume their copyright case against Google, drop it or try to negotiate a new settlement.

Paul Aiken, executive director of the Authors Guild, said in an interview that it was too early to tell what the next step would be. “The judge did expressly leave the door open for a revised settlement,” he said.

Hilary Ware, managing counsel at Google, said in a statement that the decision was “clearly disappointing,” adding: “Like many others, we believe this agreement has the potential to open up access to millions of books that are currently hard to find in the U.S. today.” The company would not comment further.

Google has already scanned some 15 million books. The entire text of books whose copyrights have expired are available through Google’s Book Search service. It shows up to 20 percent of copyrighted titles that it has licensed from publishers, and only snippets of copyrighted titles for which it has no license.

The settlement would have allowed it to go much further, making millions of out-of-print books broadly available online and selling access to them. It would have given authors and publishers new ways to earn money from digital copies of their works.

Yet the deal faced strong opposition. Among the most persistent objections, raised by the Justice Department and others, were concerns that it would have given Google exclusive rights to profit from millions of so-called orphan works, books whose rights holders are unknown or cannot be found. They also said no other company would be able to build a comparable library, leaving Google free to charge high prices for its collection. And some critics said the exclusive access to millions of books would help cement Google’s grip on the Internet search market.