The future of a proposed windfarm on Shetland touted as one of the most productive in the world has been thrown into doubt in a legal battle over a rare wading bird and an obscure part of British energy law.

A judge in Edinburgh has ruled that the Viking windfarm project, set to be one of the largest in the UK, should not have been given planning consent last year, in a victory for local anti-windfarm campaigners.

Lady Clark ruled that ministers had failed to follow an EU birds directive when they weighed up the threats posed by the project to the whimbrel, an endangered wading bird which nests almost exclusively on Shetland.

In a judicial review decision that could have serious implications for other new renewable energy projects across the UK, Clark also found against Viking Energy because the company did not yet have an electricity generating licence.

The project's collapse would be a blow to Alex Salmond's ambition for Scotland to generate all of its domestic electricity needs from renewable power by 2020. That goal is already in doubt because of the great cost and logistical problems involved.

The Viking project will comprise 103 turbines across 50 square miles of Shetland's main island. With a potential capacity of 370MW and 457MW, it is expected to produce enough electricity to power at least 175,000 homes if it goes live as planned in 2018.

Because of Shetland's powerful winds and its exposed location high up in the north Atlantic, the developers predict it will have a load factor or real-time output of at least 44% of its potential energy output. On average UK windfarms have a load factor of around 28%.

Shetland is already home to what is thought to be the world's most productive wind turbine, a small 660kw machine north of Lerwick called Betsy, which produces between 52% and 59% of its potential output. Most of Shetland's electricity, however, comes from a subsidised power station that burns diesel.

Clark said schedule 9 of the Electricity Act required a developer to have legal authority to produce electricity before it got approval. Viking Energy's co-owner is the energy giant SSE, which does have that licence. Industry sources were nonplussed, insisting it was perfectly normal for a power company to apply for an electricity generating licence after it had planning approval.

Even before her written judgment was issued, the Scottish government announced it planned to appeal against the judge's decision after hearing her verbal summary in court on Wednesday. "Scottish ministers have decided to appeal against the judgment and will lodge that appeal in the days ahead," a spokesman said.

"Ministers do not agree that the application was incompetent under schedule 9 of the Electricity Act, nor do they agree that they failed to take proper account of their obligations under the EU wild birds directive in the decision-making process."

Industry sources said they believed the government's appeal would mean that Clark's decision to withdraw planning approval would not be enforced, allowing Viking Energy to press ahead and apply for an electricity generation licence during the appeal process.

Viking Energy refused to comment, however, until the judge had published her final decision in full. That is expected on Friday.

The judge's decision will embarrass Ed Davey, the energy and climate change secretary, after he announced plans in Glasgow last month to pay a higher electricity price to boost the Viking project and other smaller onshore windfarms on Shetland, Orkney and the Western Isles.

The Viking project has been controversial since its inception. Local campaigners, Scottish Natural Heritage and the Royal Society for the Protection of Birds have twice forced a reduction of its scale, from 150 turbines down to 103.

Sustainable Shetland, which brought the judicial review, said it was delighted with the ruling. Clark also ordered the Scottish government to pay its £60,000 legal bill for the case.

The group said Scottish ministers had been warned about the legal risks over the whimbrel and the inadequacy of the developers' proposals to protect the bird. There are about 295 breeding pairs on Shetland, 95% of the UK's resident population.

Frank Hay, the group's chairman, said: "This ruling is hugely important. We have always felt that this project was wrong for Shetland – a belief upheld by our own planning department, who advised that Shetland Islands council should recommend refusal to the consent.

"We are heartened by today's outcome, which reflects months of hard work by all concerned."