The Netherlands has been ordered to pay compensation for the deaths of Bosnian Muslims in the 1995 Srebrenica massacre in a ruling that opens up the Dutch state to compensation claims from relatives of the rest of the 8,000 men and youths who died.

The judgment by the Dutch supreme court is the final decision in a protracted claim brought by relatives of three Muslim men who were expelled by Dutch soldiers from a United Nations compound during the Balkans conflict, then killed by Bosnian Serb forces.

Although the case related only to the murder of three victims, it sets the precedent that countries that provide troops for UN missions can be held responsible for their conduct.

The case was brought by Hasan Nuhanovic, an interpreter who lost his brother and father, and relatives of Rizo Mustafic, an electrician who was killed. They argued that all three men should have been protected by Dutch peacekeepers. Mustafic and Nuhanovic were employed by the Dutch, but Nuhanovic's father and brother were not.

The men were among thousands who had sought shelter in the UN compound as Bosnian Serb forces commanded by General Ratko Mladic overran the area on 11 July 1995. Two days later, the outnumbered Dutch peacekeepers bowed to pressure from Mladic's troops and forced thousands of Muslim families out of the compound.

Bosnian Serb forces sorted the Muslims by gender, then began killing Muslim men and boys. The bodies of approximately 8,000 were buried in hastily dug mass graves.

The international criminal tribunal for the former Yugoslavia (ICTY) in the Hague has ruled that the killings constituted genocide and Mladic is on trial for crimes committed at Srebrenica. The atrocity was the worst massacre on European soil since the second world war.

The Dutch court ruling held that in the chaos of the Serb takeover of Srebrenica, UN commanders no longer had control of the troops on the ground and "effective control" therefore reverted to Dutch authorities in the Hague.

The human rights lawyer Liesbeth Zegveld, who represented the Bosnian families, called the ruling historic because it established that countries involved in UN missions can be found legally responsible for crimes, despite the UN's far-reaching immunity from prosecution. "People participating in UN missions are not always covered by the UN flag," she said.

Toon Heisterkamp, a supreme court judge who briefed the media, insisted that the narrow focus of the case meant it was unlikely to have far-reaching effects.

Outside the courtroom, Nuhanovic said he was stunned by the ruling, which ends a 10-year legal battle and opens the door to compensation claims against the Dutch government.

"I was thinking about my family, they are dead for 18 years," he said. "It does not change that, but maybe there is some justice. It should have happened years ago. In the future, countries might act differently in peacekeeping missions and I hope the lives of other people in the future will be saved because this mistake was admitted."

The Dutch government resigned in 2002 after the National War Documentation Institute blamed the debacle on Dutch authorities and the UN for sending underarmed and underprepared forces into the mission and refusing to answer the commanders' call for air support.

The government accepted "political responsibility" for the mission's failure and contributes aid to Bosnia, much of which is earmarked for rebuilding in Srebrenica. But it has always said responsibility for the massacre itself lies with the Bosnian Serbs.

The three men were among the last to be expelled, the 2011 ruling said, and by that time the peacekeepers, known as "Dutchbat" for Dutch battalion, had already seen Bosnian Serb troops abusing Muslim men and boys and should have known they faced the real threat of being killed.

"Dutchbat should not have turned these men over to the Serbs," a summary of the judgment said.

The Hague appeals court in 2011 ordered the families of the three dead men to be compensated, but no figure was ever reached, pending the outcome of the government's appeal to the supreme court.

Zegveld said the amount of compensation the families will receive was not important. "It's far more important what's been decided today than any amount that will be established in the future," she said.

The Srebrenica massacre has turned into a national trauma for the Netherlands. Dutch troops returning home faced accusations of cowardice and incompetence.

The Dutch soldiers, many of whom feared for their own lives, helped the attacking Bosnian-Serb troops as they separated Muslim men from women. The men and boys were then bussed to execution sites. A subsequent inquiry exonerated the ground forces.

"Dutchbat decided not to evacuate them along with the battalion and instead sent them away from the compound," a summary of the supreme court ruling said.

"Outside the compound they were murdered by the Bosnian-Serb army or related paramilitary groups."