“People were suspicious of each other, but they came together, here in Cable Street.” The speaker was Leon Silver, the president of the East London Central Synagogue, and he was addressing a small crowd in the evening sun. “Irish dockers, who at the time were quite territorial, came out and dug up paving stones. Christians, Jews, communists, socialists, even people from the right who were not fascists, came out. We were united.”

Silver was speaking on the 80th anniversary of the Battle of Cable Street, as it is known in left-wing mythology, when East End residents blocked a march by the British Union of Fascists, and it became a street fight. The residents won – and the fascists retreated. On the building behind Silver, a fiery mural captured the heat and passion of the battle.

Cable Street lies in Tower Hamlets, a London borough still known for both its diverse immigrant communities, and its poverty. That evening, though, it seemed like a tranquil place. Cyclists whizzed by on their way home from Canary Wharf. A multi-cultural band played upbeat tunes. A cynic might ask whether the commemoration is even necessary.

Yet Tower Hamlets has not been immune to the effects of the Brexit vote. The mayor, John Biggs, told me: “We have had quite an increase in reported hate crimes in terms of verbal crime, nothing in terms of physical acts of violence, but we are on our guard.”

In 1936, the residents of Cable Street might have been opposing the fascists, but their physical fight was with the police, who were enforcing the fascists’ right to march. Oswald Mosley, the fascist leader, was an establishment politician who served both the Conservatives, and then Labour.

Today’s politicians are, unlike Mosley, quick to condemn hate crimes. But this is not enough. The language around immigration has grown more and more charged since the EU referendum, even as it stays within the boundaries of middle-class respectibility. On the dawn of Brexit, the then-Ukip leader Nigel Farage called the vote as “a victory for decent people”. The sentence on its own sounds innocuous. It was the context that mattered.

This polarising language has crept into the Conservative party conference. The home secretary Amber Rudd briefly praised the "many benefits" of immigrants but devoted far more time to singling out problems with them, including foreign students who weren't "proficient" in English, and illegal immigrants who had to resort to sleeping rough on the streets.

Meanwhile, the health secretary Jeremy Hunt, who is facing mutinous junior doctors and a gaping NHS funding crisis, found the space to call for more "home grown" medics. Each statement on its own seems reasonable. Instead, it is the repeated focus and prioritisiation of immigration that draws a clear dividing line between "them" and "us".

“There are good economic reasons for managing flows across borders,” said Biggs. “Of course there are.

“But the problem with the arguments exploited by Brexiteers is they include a nod and a wink that ‘your problems are because of these other people in the country’.” He paraphrased the words of Justice Oliver Wendell Holmes, a US lawyer: “Does freedom of speech allow you to cry fire in a theatre?”

He also criticised Labour for its sluggish response to incidents of anti-Semitism: “I think one of the risks with the left is, while they are not all Trots and Marxists, the thing that informs them is class. They ignore the subtleties.

“When people from Momentum say they don’t understand the arguments, they genuinely don’t. And this means we have a job to educate them about the perniciousness of divisive politics.”

Perhaps the Conservatives’ relentless focus on immigrants would feel more innocent, and Labour’s anti-Semitism debate more trivial, if it wasn’t for history. Today’s East Enders may celebrate the Battle of Cable Street, but it was followed by the Mile End Pogrom, when fascists attacked anyone visibly Jewish and a young girl was reportedly thrown through a glass window. During the war, when hundreds of Londoners were crushed to death as they tried to shelter in Bethnal Green tube, some locals blamed the tragedy on "frightened" Jews. Meanwhile, in Germany, the Nazis were embarking on the Final Solution.

There are legitimate debates to be had on both left and right about freedom of speech, the link between economics and immigration, and how to deal with Brexit. But it is impossible to do so, without acknowledging the shadow of history on our present. Cable Street may not be the frontline anymore, but so long as EU migrants feel scared to speak their own language on the street, and hate crimes rise, the battle is still going on.

“I think it is the duty of one generation to teach the next one about the past,” said Biggs. “We learn the lessons of what has happened.” Let’s hope.