Some authors had complained they had not given permission for books to be scanned and made available online

Google's controversial plans to create the world's biggest online library have been shelved by a US judge.

In a ruling filed in the US district court in Manhattan, judge Denny Chin ruled the company had gone "too far" in its ambitious plans and rejected a legal settlement with authors and publishers that Google reached in 2008.

The web giant has scanned millions of books, many held at some of the world's greatest libraries including Oxford University's Bodleian and Harvard's libraries, and made them available online via its eBooks platform. The plan has split the publishing industry and attracted fierce criticism from authors and rival tech firms.

While Google said it would show only snippets of works that are in copyright, some authors complained that they had not given their permission for the scanning in the first place and were wary of Google's future plans.

In court Google rejected calls for an "opt-in" solution where copyright owners would decide whether or not to be part of the scanning project. The company said the idea was not viable. Chin suggested he might look more favourably on a settlement that allowed copyright owners to "opt in".

"While the digitisation of books and the creation of a universal digital library would benefit many," Chin wrote, Google's current pact would "simply go too far". It would "give Google a significant advantage over competitors, rewarding it for engaging in wholesale copying of copyrighted works without permission," he said.

The agreement rejected by Chin was negotiated with the Authors Guild and the Association of American Publishers. Under the settlement, Google would continue to digitise books and sell access online and the company would pay $125m (£76.9m) in royalties every year to the copyright owners of the books being scanned.

Copyright concerns persisted, however, as the ownership of many of the works being scanned by Google could not be established.

Hilary Ware, managing counsel for Google, said the judge's decision was disappointing. "We believe this agreement has the potential to open up access to millions of books that are currently hard to find in the US today," she said. "Regardless of the outcome, we'll continue to work to make more of the world's books discoverable online through Google Books and Google eBooks."

"Publishers are prepared to modify the settlement agreement to gain approval," said John Sargent, chief executive of Macmillan, in a statement issued by the Association of American Publishers. He said they would work to overcome the objections raised by the court.

Google co-founder Larry Page was the author of the firm's plans to make 150m books accessible via the search engine. He has been promoting the idea since shortly after the company was formed in 1998.

Google began working with several libraries in 2004 to scan and digitise books and other writings in their collections, and has said it has completed 10% of the effort. The search engine currently allows users to search about two million books that are out of copyright, including the works of William Shakespeare. That service will be unaffected by the ruling.

Chin's decision is the latest in a series of setbacks. The plans have attracted criticism not only in the US but across Europe and in China and Canada. It is also separately being investigated by the US Justice Department on competition and copyright grounds.