Every week for the past four months, officials at the fortress-like British high commission in Kingston have been comparing lists of names of potential Windrush victims with lists held by staff a mile away at the Jamaican foreign ministry.

Finally their lists match and officials are working on the cases of 13 people whom the British government believes it wrongly deported to Jamaica. Eight of them have been contacted, two remain untraced as yet, and three are believed to have died before officials were able to reverse the wrongful removals.

Officials are also dealing with a bigger group of Windrush people who found themselves stuck in the Caribbean after travelling there for funerals or holidays and were refused re-entry to the UK. Some have been issued with documents allowing them to travel back on “returning residents” visas.

But there is a third and possibly much larger category of people that the government no longer wants to highlight: those who were wrongly deported after committing a criminal offence.

David Jameson believes he may be in that category. He is currently sleeping rough in Kingston, waiting for the error to be rectified. It may take longer for officials to focus on his case because before he was deported he was convicted of a relatively minor offence related to the London riots and sentenced to 14 months in prison.

In his initial acknowledgement of wrongful deportations, Sajid Javid said 63 members of the Windrush generation could have been wrongfully removed, 32 of whom were labelled foreign national offenders. In his subsequent update on the numbers of wrongly deported people, the UK home secretary said after a new analysis of files that the number of people who had been mistakenly removed from the UK had gone up to 83, but he said he was no longer including foreign national offenders in the count and he was making a “purposeful distinction between criminal and other cases”. Once people with convictions are included, the total number of wrongful deportations is likely to be much larger.

Jameson, whose real name is being withheld because of his embarrassment about his criminal conviction, was deported on 5 June 2013. Because he had tried to kill himself twice in the detention centre while he awaited deportation, he was accompanied on the British Airways flight by two medical assistants and two guards. He was handcuffed all the time.

“I was very calm, because that is what the medication does, but I was also really scared,” he says. He was driven straight from the airport in Jamaica to a mental health facility and left there.

After a lifetime out of the country he had no ties in Jamaica. As a child in the UK he remembers being abused in the streets. “I was called black monkey, asked if I came on the banana boat, and told go back home,” he says. Back in Jamaica he is seen as an alien. “I’ve been called names – deportee, foreign. I’ve been threatened. I often feel frightened.”

David Jameson: ‘I don’t know where I’m going to sleep tonight.’ Photograph: David Levene/The Guardian

For a while he has been staying in a derelict, roofless Police Federation building in downtown Kingston, sleeping on a bit of cardboard next to piles of rubble, broken glass and rotting rafters. “I don’t know where I’m going to sleep tonight. I don’t have stable meals, I sleep in very disgusting places, old buildings, abandoned shops and houses.” He describes Kingston as a very violent place and says he witnessed a number of murders while living on the streets. “They have deported me to a place which is a war zone.”

Since being deported he has lost his flat in London, and the contents have been thrown away. He keeps all his Home Office information in a black plastic bag, which he carries with him. He is struggling to retain contact with his girlfriend and two teenage stepdaughters who had him as a father figure for nine years. He tries to speak to them a couple of times a week. “I miss them. I keep up with their news.”

Jameson, 58, says he was brought to London when he was six in 1966 by his grandmother, travelling on her passport, because his mother in Jamaica was unable to look after him. He says he realised at about the age of eight that his grandmother was an alcoholic and was also unable to care for him properly.

He had an unhappy and chaotic childhood. When his grandmother died, the contents of her flat were thrown away and as a result Jameson says he has been unable to supply any evidence of having lived in the UK before 1973 (during the period when Caribbean migrants had the right to arrive without a visa and attain indefinite leave to remain).

For a long time he was employed informally because he had no papers. He says he helped stall owners at Brixton market, worked on demolition sites stacking bricks for resale, and worked in Paddington station on night shifts cleaning toilets. When he was issued with a temporary national insurance number he worked for Willmott Dixon on building projects in Canary Wharf. After about 10 years with the firm he was sacked because he was unable to provide the right documents to get a permanent NI card. A couple of times he tried to resolve his immigration status but was unable to afford the fees quoted by lawyers.

Desmond Johnson: ‘I just want to see my children.’ Photograph: David Levene/The Guardian

In Harmondsworth detention centre he told officials he had been in the UK for 46 years and they asked for proof. From inside the immigration removal centre he was unable to gather any evidence of long-term residency in the UK, except for a letter from a Brixton community leader saying he had known him as a talented and helpful member of a youth club in 1985. This was not enough.

Despite Javid’s remarks about people with convictions, there appears to have been some recognition already that a mistake may have been made in Jameson’s case. He had a phone call from someone at the Windrush taskforce 10 days ago asking him where he would stay in the UK and if he had enough money for a fare back. “She was very nice. She kept using the term ‘there is light at the end of the tunnel’,” he says. But he has heard nothing since and is uncertain about whether he will be given permission to return. “It is extremely painful. My life is in ruins and I am on the edge. Sometimes I want to go down to the sea and jump in and never come out.”

The decision to exclude people who have committed offences from active assistance has been controversial. Celia Clarke, director of the charity Bail for Immigration Detainees, which advised Jameson while he was in detention, said: “It is completely unjust that the Home Office should differentiate between long-term British residents with criminal records and those without. If they have spent their childhood and subsequent years in the UK then they are British, whether or not they have the requisite piece of paper that confirms this. It is completely unacceptable, disproportionate and unspeakably cruel to send people into what amounts to permanent exile to satisfy some perceived need to be tough on immigration and crime.”

Diane Abbott, the shadow home secretary, said: “The individuals that this government has deported to the Caribbean are first and foremost British citizens, and as a country we do not deport our citizens. We must have full transparency about the scale of the Windrush scandal and the government must not attempt to cover up the total number of our fellow citizens who have been deported, detained and denied their rights.”

A Home Office spokesperson said: “We do not routinely comment on individual cases.” The department said it was the home secretary’s “duty to protect the public”.

Of those people who were wrongly refused re-entry to the UK after visits to Jamaica, some have already successfully made contact with the taskforce and been issued with visas to return.

Desmond Johnson, 59, who arrived in the UK aged 11 and worked in construction in London until he returned to Jamaica to support his mother after his father’s funeral, has no desire to go back permanently to the UK because he has made Jamaica his home. However he is delighted that he will now be able to visit his daughter whom he has not seen in 16 years, and relieved to know he will be able to go back if anyone in his family becomes unwell. “I’m happy that the visa has been granted. I don’t want to live there. I just want to see my children,” he says.

Ken Morgan: ‘The British ... will just drag this out until we die.’ Photograph: David Levene/The Guardian

Ken Morgan is more circumspect. His British passport was confiscated in the mid-1990s when he tried to return to the UK after a family funeral. He had arrived in the UK at the age of nine as a British subject in 1959, before Jamaica became independent, but after the funeral he was unable to persuade officials at the British high commission he was British. Since that shattering moment he has successfully rebuilt his life in Jamaica, first opening a shop and then working in graphic design at the University of West Indies.

When the Windrush scandal broke in April, he suddenly began to get “a ton of calls” from the high commission. After spending decades trying in vain to get someone there to listen to him, he thinks he was called about 20 times in the space of a fortnight.

He has been granted a returning resident visa but is dismayed at the secretive way he feels he has been treated by British officials. “When they call, I ask: ‘Who am I speaking with?’ and they say: ‘I’m not allowed to tell you.’ So many different women would call, saying: ‘Mr Morgan, we just want to verify something more.’ They’re nice and courteous but it is a fortress mentality – they won’t tell you who you’re speaking to.”

He is not holding out hope for compensation. “The British know that we are old people, we’re not young and sprightly. They will just drag this out until we die.”

All over Kingston, anger about the British government’s treatment of some long-term residents has not abated. Herbie Miller, director of the Jamaican music museum, gave a furious interview at the entrance to a Jamaican 1940s music event, near the seafront. “So many of us are aware that our parents’ generation migrated to the so-called mother country, as colonial subjects. Many fought in two world wars for the mother country, and then following the devastation in Britain they were allowed to go back to rebuild Britain. To see how the British government treated people who grew up in England who came home for a funeral or a vacation or a wedding, and were suddenly told: you’re an illegal immigrant ... it hurts.

“These people worked in your hospitals, your transport systems, they took low-paid menials jobs,” he said. “It hurts because these people spent their life there, they went as little kids, and then they are put through this anguish. Britain should be ashamed of itself.”

Bert Samuels: ‘We fought in two world wars, we sent our soldiers who shed their blood for Britain.’ Photograph: David Levene/The Guardian

Bert Samuels, a barrister who sits on the National Council of Reparations, campaigning for the UK government to pay compensation to the descendants of slaves in Jamaica, says the Windrush scandal is closely connected to the call for reparations for the slave trade.

“The legacy of slavery is why we are so impoverished and why so many of us have had to leave Jamaica for greener pastures, to send remittances home,” he says. “We were British subjects until 1962. We fought in two world wars, we sent our soldiers who shed their blood for Britain. Then all of a sudden it became a policy that we had to apply for a visa to go to a country that used us for three centuries. We felt discarded. It is widely accepted in Jamaica that Britain has used us and refused us.”