High on a hill above Bolton on a sunny Sunday in June, a mother and three children were rushing for a bus on their way to church when they encountered an angry local man, Dale Hart.

They had not been in Bolton long, having arrived on the UN’s Gateway Protection programme, which offers a legal route for up to 750 long-term refugees to settle in the UK each year via camps in Africa and the Middle East.

The children – a boy aged 15 and two girls, 13 and 15 – had not yet learned much English, which proved a problem when the boy had to call an ambulance and explain where his mother had collapsed after Hart hit her.

Most local authorities don’t accept any Gateway refugees. But since the scheme started in 2004, Bolton has welcomed 2,307, including 255 in the last year alone.

The town is also home to a disproportionate number of asylum seekers. Migrants who come to the UK seeking sanctuary are then dispersed to the cheapest parts of the country via a government scheme while they await refugee status. At the last count, in the third quarter of this year, there were 1,012 asylum seekers in Bolton, up from 563 a decade ago and more than in any other UK town. Only the cities of Glasgow, Cardiff, Birmingham, Liverpool and Manchester have larger populations of asylum seekers.

The issue has united Conservatives and Labour on the local council. Labour’s Linda Thomas, the council leader, said this week that while Bolton had “a long and proud history [as] a haven to those that have suffered persecution … our asylum-seeker figures continue to be disproportionately high … and other authorities need to do their fair share.”

Last month, Andy Burnham, the mayor of Greater Manchester, threatened to withdraw the region from the dispersal system. “It cannot be right that towns in Greater Manchester have more asylum seekers clustered in a handful of wards than entire regions in the rest of the country,” he said.

David Greenhalgh, leader of Bolton’s Conservative group, said that not only did Bolton take more than its fair share, the arrivals were too often placed on “deprived estates with nationalist tendencies”.

Last week in Bolton, a court heard how the mother ended up in hospital for three weeks after being punched by Hart, a 29-year-old customer service adviser who lived near her in Breightmet, a working-class suburb. He hadn’t meant to hit her, his barrister explained. He was actually swinging for her son: he wrongly thought the teenager had barged his son’s buggy.

Hart admitted he told the boy to “speak English” and then, during what his barrister called “an unsightly scuffle”, shouted: “Get off me, you black cunt.”

The case made headlines when police briefed that the mother was seriously ill in hospital with two haematomas – bleeds on the brain. Hart was caught and charged with racially aggravated grievous bodily harm. Within days, an anti-racism march was organised in Breightmet, the location of many of the 83 houses Bolton reserves for Gateway refugees.

The attack polarised the community, according to Jon Lord, chief executive of Bolton at Home, the housing association that owns most of the properties. He opposed the march, which he said had made his job harder “in smoothing over some of the pain in the community”.

Figures released to the Guardian by Greater Manchester police (GMP) under the Freedom of Information Act show that race hate crime in Bolton has increased significantly. In June 2013, GMP recorded 21 reports of race hate crime. In June this year the figure was 90.

Facebook Twitter Pinterest Top o’ th’ Brow in Bolton. Photograph: Christopher Thomond/The Guardian

Some refugees and asylum seekers say they are happy in Bolton, but some experience abuse, said Shaheda Mangerah, a case worker at Bolton’s Destitution Project. “Two Iranian ladies, asylum seekers, told me that they were having a picnic in the park with their kids when someone came up and said: ‘Why are you here? Go home. Just because you have dyed your hair blonde doesn’t mean you will fit in.’”

GMP said the rise in reports of hate crime in Bolton was due “in part to an increased confidence in victims coming forward, but also because of changes to the way that forces record such crime”. Over the last 12 months, 52 reports for a racially motivated hate crime in Bolton have resulted in a charge, summons, caution or restorative justice or other community resolution, according to Ch Insp Chris Bridge.

Local people who understand the complicated system of refugee and asylum dispersal say the situation worsened dramatically when the government outsourced the asylum contract to Serco. Until 2011, the council decided where to place the newcomers, and officers could ensure arrivals were housed in small clusters rather than “swamping” streets, said Lord.

He claimed the council made a surplus on the old asylum accommodation contract and was able to spend the extra money on teaching assistants, social care costs, health workers and extra nursery places.

Serco currently has around 300 private houses in Bolton for asylum seekers, and has little compulsion to mitigate the impact of its tenants. To make a profit, it chooses the cheapest accommodation. Until recently, there has been minimal dialogue with the local authority, although the council said it occasionally objected when Serco applied to use a new property for asylum seekers in an unsuitable area.

Now Boltonians sometimes complain that refugees and asylum seekers put them at a disadvantage. In Breightmet last week, Nicola Barlow, 32, said she was living in a three-bedroom house with six children “and another on the way” and was still at 13 on the housing waiting list. Meanwhile, she saw refugees and asylum seekers move straight into bigger houses. At her son’s school, teachers had to spend time trying to teach new children English. She said: “Our kids don’t end up doing as well as they should.”

Lord said the school complaint was fair. With budgets under pressure, it is perhaps inevitable that teachers struggle to cater for all pupils. One of the reasons Bolton council still takes part in the Gateway project, according to Lord, is that it at least comes with some government money attached, which pays for a specialist nursing service for asylum seekers and refugees that offers a TB screening service.

At Bolton crown court last week, Hart’s barrister reached a plea deal with the prosecution. The GBH charge was dropped and he was allowed to plead guilty to racially aggravated assault and affray. The judge accepted that his victim had a pre-existing kidney condition that meant she was at increased risk of haematoma. She had made a good recovery, the court heard. The family has been moved out of Breightmet and are happy, according to those who know them.

Hart was given a 12-month suspended sentence, along with 18 hours of community service and 30 days of rehabilitation. He would have gone straight to jail if the prosecution had proved the initial attack was motivated by racial hatred, the judge said.

Meanwhile, asylum seekers continue to arrive in Bolton. Serco knows about the Hart case and wants to avoid it happening again.

Jenni Halliday, a contract director for Serco Compass, said: “We work closely with local authorities across north-west England to find suitable accommodation for the asylum seekers in our care and their safety is very important to us … in Bolton neither the number of asylum seekers nor the number of properties has risen since 2016 and we have no plans to increase them.”