Six teenage boys have been arrested in Harlow, Essex, after an eastern European immigrant died following a brutal street attack which his brother said began after he was heard speaking Polish in the street.

A murder investigation was launched on Monday into the death of Arkadiusz Jóźwik, 40, from Poland. The factory worker, who had been living in England for the last four years, was knocked unconscious outside a row of takeaway shops on Saturday and later died in hospital.

Essex police, who were called to the scene shortly after 11.35pm on Saturday, said the attack on Jóźwik and a second Polish man who survived the assault was apparently unprovoked and that one line of inquiry was the possibility of it being a hate crime.

During an emotional visit on Tuesday to the scene of the attack, Jóźwik’s brother, Radek, spoke of how there had been a surge in hatred towards eastern Europeans after Britain’s vote to leave the European Union. He said he been told by police that the attack happened after they heard him and his friends speaking the Polish language.

“He was standing eating pizza and they picked on him because of that. He does not speak much English. The young teenagers are so aggressive.

“Parents need to speak with their children – they are out of control. One of the teenagers started and then seconds later there are 10 or 12 people attacking him.

“After the Brexit vote it has got worse – I have seen people change – it is hard at the moment.”

Four 15-year-old boys and one 16-year-old boy, all from Harlow, were arrested on Monday on suspicion of attempted murder and then arrested on suspicion of murder. All five have been bailed until 7 October pending further inquiries. A sixth boy, a 15-year-old from Harlow, was arrested on Tuesday afternoon on suspicion of murder and is awaiting questioning.

The attack took place outside some shops in an area of Harlow known as The Stow. Jóźwik suffered head injuries and was taken to the Princess Alexandra hospital in Harlow. He was later transferred to Addenbrooke’s hospital in Cambridge, where he died on Monday.

The 43-year-old man who was attacked at the same time was treated in hospital for suspected fractures to his hands and bruising to his stomach.

The senior investigating officer, DI Al Pitcher, of the Kent and Essex serious crime directorate, said: “This is now a murder investigation and our inquiries have quickly led to the arrest of five teenage boys. Following this vicious attack, a man has sadly lost his life and I urge anyone with information to contact us as soon as possible.”

Police will be carrying out extra patrols in the area to provide reassurance to the community.

Some local people who declined to give their name sought to play down allegations of hate crime. “There has been a problem for a while now of young people gathering here and staying late. I find it intimidating to walk past them, and I’m saying that even as a teacher,” said one woman.

“There has been a group gathering here for some time on a regular basis and just recently I’ve seen some new faces, which have made it a bit different. Where are the parents of these young people and do they know where their children are at this time of night?”

Some residents have said the attack was not the first instance of physical violence against Polish people in Harlow since the Brexit vote.

Polish journalists said that they had spoken to a number of people who said two Polish men were taken to hospital several weeks ago following an attack in a street near to where Jóźwik was assaulted.

There has been a rise in hate crimes following the vote on 23 June. More than 3,000 allegations of hate crimes were made to UK police, mainly in the form of harassment and threats, in the week before and the week after the vote, a year-on-year increase of 42%.

A UN organisation said last week that British politicians had helped fuel a steep rise in racist hate crimes during and after the referendum campaign. The committee on the elimination of racial discrimination said many prominent politicians should share the blame for the outbreak of xenophobia and intimidation against ethnic minorities.

Jóźwik’s brother said their mother, who lives in the UK, had come back from holiday to the news of her son’s death. “We had to meet her at Stansted airport and tell her and then take her straight to the hospital. She is really struggling,” he said.

“Arek came down to get a pizza – it was the first time he came down to get one. Normally he would get it delivered but for whatever reason he came to buy one and that’s when all the trouble started.”

Radek said he had lived in Harlow for the past 12 years and that the killing would not force him to leave the country he now regards as home. “I had said to him, ‘Come on Arek, come over, I will help you find a job. He wanted to stay here.

“We will not leave because of his though. I have two children and we have British passports. This is my home.”

Polish people who spoke to the Guardian on Tuesday night at the scene of the attack said tensions had been building in the town since the Brexit vote.

One man, who worked with Jóźwik, said he and others had increasingly been on the receiving end of abuse from some residents.

“I’m not afraid of these people though,” said one young man. “I won’t be going home. I can defend myself.”

He said a tearful minute’s silence had been observed at their workplace earlier in the day.

“Since the vote, people have been saying things to us for some time. It’s been getting more difficult.”

People from Harlow had left tributes and flowers at the scene, where Polish journalists carried out interviews for what they said was the main news story at home.

The Polish embassy said its representatives in the UK were investigating the incident and had visited Harlow to speak to police.

Wes Streeting, the Labour MP for Ilford North, said on Twitter that the attack “brings shame on our country”, while the leader of Harlow council, Jon Clempner, said that whatever the motive for the attack, it would never have a place in the town.

“I know that Harlow and its communities will support each other through this and that all our communities will stand together.”