The British government is accused of presiding over the emptying of the remote colonial outpost of Ascension Island in the mid-Atlantic, uprooting families that have lived there for almost a century. Local people claim the UK is engaged in a slow-motion repeat of its widely condemned expulsion of the inhabitants of the Indian Ocean island of Diego Garcia.

Ascension Island is a tiny leftover of empire, a volcanic island 700 miles from anywhere. It is dominated by a US military airbase, which more than 100 aircraft passed through on security duties during Barack Obama's recent trip to Africa. There are satellite and submarine tracking stations, a BBC transmitter, and a listening post run by GCHQ's Composite Signals Organisation.

Its resident population – most of them originally from St Helena, another British South Atlantic island – has fallen by a quarter in a decade to less than 800, as the companies that now run most military and civilian services replace settled family communities with contract workers. Local people say the island now has more antennas than people.

Caroline Yon, a former island councillor whose day job is running a European Space Agency tracking station, said: "The US and UK are squeezing the life out of the place. They want to make Ascension like Diego Garcia." Britain expelled the population of Diego Garcia in the Indian Ocean 40 years ago to make way for a US airbase.

A shopkeeper, Cedric Henry, said: "It's like being on an oil rig now. We have no rights. We are just a workforce, even though many people have never lived anywhere else. Some families have been here for four generations."

The issue is expected to come to a head in elections later this year for the island council – a purely advisory body that is the island's only semblance of democracy after 198 years of British rule.

It wasn't supposed to be like this. A decade ago, in the heyday of Labour foreign secretary Robin Cook's "ethical foreign policy", Britain promised a new deal for the residents. It drew up plans for democratic institutions, a legal right of abode and to own property. "We want to ensure that Ascension continues to be a viable community," said the island's then administrator, Andrew Kettlewell.

The new island council planned to develop eco-tourism. The only downside was the introduction of taxes.

Then in 2006, Cook's successor, Jack Straw, committed what the island's administrator, Colin Wells, admits was a "spectacular U-turn". The promises were all abandoned, though not the taxes.

With no right of abode, anyone who retires or reaches 18 without a job, or whose contract ends, has to leave. Businesses set up during the "Ascension spring" have lost their value because they cannot be sold and have no secure land tenure.

Over afternoon tea in the garden of his official residence, guarded by ceremonial cannons, Wells denied local claims that Whitehall was forced into the about-face on right of abode by American security fears about potentially troublesome neighbours. Ministers wanted to avoid "contingent liabilities" such as providing pensions, unemployment pay and beefed up security, he said. "It was a necessary U-turn."

But many St Helenians, known on the island as "Saints", feel frozen out, said the island's internationally regarded conservation officer, Stedson Stroud, himself a Saint. They have been losing professional jobs, such as teaching at the island's only school, to Britons on short contracts. Stroud, who retires soon, fears he will be replaced by a young British scientist rather than his able local deputy.

Stroud added that British activities had left the island without its own natural produce. "Since the farm on the island closed, all the food is imported. Most of the fresh fruit and vegetables are handouts from the US base."

Last month, a former Ascension councillor, Lawson Henry, who returned to St Helena in disgust at the British U-turn, said Saints forced back were discriminated against twice over by the British authorities, because they got no pension on St Helena. "A person must work on St Helena for a minimum of 20 years [to qualify]. This effectively cuts out all those St Helenians who have worked on Ascension for all or most of their working lives."

The Saints are mixed-race descendants of European colonists and the African slaves and Asian labourers brought to St Helena to work on flax plantations to make mailbags for the Post Office. They began moving to Ascension in the 1920s to work in short-lived guano mines. They now feel surplus to requirements, an unwanted legacy of empire.

The biggest cause of Ascension's recent depopulation, said Yon, was the privatisation of most government and military services on the island.

The British contractor Interserve is now the island's biggest employer. Jobs are being shed and workers moved on to short-term contracts. Families now only accompany workers if that is essential to fill positions, say officials. The headteacher was allowed to bring his wife and two children from Britain.

The result is that "the population is shrinking to a critical mass", said Yon. The school could be an early casualty. It has about 100 pupils, educated up to 16 years, but Wells said a roll anywhere below 75 would be unviable. The loss of families means that three-quarters of the population is now male. Sexual exploitation of the remaining teenage girls is becoming a serious problem, said Yon.

Since 2006, in response to questions about the rights of islanders, ministers have insisted that "there is no indigenous population, or 'islanders'", Wells said: "On Ascension, everyone is an expat, present by virtue of an employment contract."

Stroud said many Saints believed Britain planned to abandon Ascension, evacuate the resident Saints, and leave it to the Americans – "perhaps when the airport is completed on St Helena in 2016". That would give the RAF an alternative refuelling stop en route to the Falkland Islands.

Wells denied this and rejected the comparison to Diego Garcia. "The British Indian Ocean territory and Ascension remain very different," he said.

But the promise of a decade ago that the residents of Ascension could forge a permanent presence on the island is becoming a distant dream.