Thousands of former residents of the Chagos Islands, who were forcibly removed by Britain to make way for a US military base, have lost their latest legal bid to return home at the UK's highest court.

Judges at the UK's Supreme Court dismissed the islanders' appeal by a majority of three to two on Wednesday, dashing their hopes of returning to the tropical paradise they were forced to flee exactly 50 years ago.

More than 2,000 people once lived on the islands in the Indian Ocean, which had thriving villages, a hospital, a church, and a railway, before Britain in 1966 leased the main island of Diego Garcia - then part of British-controlled Mauritius - for 50 years to the United States in return for a discount on buying Polaris nuclear missiles.

Today, Diego Garcia is believed to be one of the largest US military bases in the world, and home to an estimated 4,000 US troops.

Richard Gifford, a lawyer representing the Chagossians, told Al Jazeera that the battle for the islands' former inhabitants to return home was far from over.

READ MORE: Chagos: The heart of an American empire?

"In reality all five judges have said, in different ways that they should never have had their right of abode taken from them, and that should all now be reconsidered by the government."

The judges highlighted a study commissioned by the government in 2015 that concluded that it would be possible and affordable to return the islanders.

Olivier Bancoult, the President of the Chagos Refugees Group and the man who brought the legal challenge, told Al Jazeera that the ruling highlighted the need for the UK to "shoulder its responsibilities."

"There is a very strong signal given to the UK government to shoulder its responsibilities towards the Chagossians. I think we'll move now to the next direction, a political way which is approaching the prime minister to ask him to make as soon as possible the resettlement taking place on Chagos," Bancoult said.

Hundreds of families were forced to leave the archipelago in the early 1960s, with many Chagossians herded into cargo boats and ditched in Mauritius and the Seychelles, where they were forced to live in dilapidated shacks and slums.

READ MORE: UK govt pressed over allegations CIA used Diego Garcia as black site

In 2000, the UK's High Court ruled that the islanders could return to 65 of the islands, but not to the main island of Diego Garcia. But in 2004, the government nullified the High Court's decision by invoking the royal prerogative. But this in turn was overturned three years later when judges rejected the government’s argument that the royal prerogative was immune from scrutiny.

In 2008, the government won an appeal in the House of Lords, which ruled that the exiles could not return.

In recent years, the US has been accused of using the base, whose lease expires this year, for acts of illegal rendition during the so-called War on Terror.

In 2014, sources told Al Jazeera that British authorities were in "full cooperation" with US intelligence to use the island as a "black site" prison to detain high-value suspects, secretly operated by the CIA.

Since 9/11, CIA black sites have been set up all over the world, with suspected terrorists subject to brutal abuses including sleep deprivation, sensory deprivation, auditory overload, rectal rehydration, waterboarding and stress positions, as well as other forms of treatment designed to humiliate and degrade.