The Brexit result split those who support it. The darker politics that was mixed in with the mainstream leave vote no longer feels the need to hide. On Monday, Anna Soubry, Owen Jones and I were confronted by it. But those who verbally attacked and physically harassed us do not speak for ordinary leave voters or the disadvantaged. They have nothing to do with the pro-Brexit protesters, with whom I respectfully disagree.

I’ve encountered them before; talking to people around the country is what I do. I’ve engaged with them at People’s Vote marches in Sunderland and outside parliament. I even attended a rally led by the English Defence League’s so-called Tommy Robinson. A figure close to Robinson admitted to me (believing I was pro Brexit) that they knew Brexit would make the country poorer. Knowingly going ahead with a policy that leaves desperately squeezed people even worse off is a betrayal of those they claim to represent.

These far-right groups go to the most neglected parts of the country, like Sunderland, and prey on existing frustrations with a London-centric political system that has left them behind. And instead of pushing to address regional inequality and the democratic powerlessness created by our constituency-based voting system, they point the finger at minorities and the EU. And now that leaving the EU hasn’t worked out, they seek to silence any national dialogue about what to do about it.

I’m not afraid of these sad few men in their hi-vis jackets, ironically copying a movement that started in France. But I am afraid they will define what constitutes acceptable debate around Brexit. It would be a disaster if we let the shape of UK democracy and dialogue be fixed by the demands of thugs.

Bercow urges Met chief to take action against protesters abusing MPs Read more

The abuse and threatening behaviour of these individuals around parliament is unacceptable. And it isn’t just heckling. In this instance, a group of men surrounded an MP. Some individuals in that group aggressively shouted abuse in her face, then appeared to physically block her from entering parliament (her place of work).

Plenty of people around the country are frustrated with the Brexit process – and all but a tiny minority continue to engage in reasonable democratic debate. This debate is absolutely crucial given the vastly different and country-defining options the UK faces. We must be able to have it without feeling like some options are unacceptable because a tiny minority of people are willing to go further, and shout louder, than everyone else. The answer to the far right cannot be to give in. We have never, and will never, let them dictate the terms of debate. When they argue for less democracy, we must call for more.

I disagree vehemently with Jacob Rees-Mogg. But when far-left activists went to his house and verbally attacked him in front of his children, that was wrong. When Julia Hartley-Brewer, with whom I regularly argue on the radio, receives abuse on Twitter, I defend her. Because there’s a key difference between intense political discourse and gratuitous abuse, no matter where you are on the spectrum.

That’s why I’m constantly calling out those on my side who label all Brexit voters uneducated, as if remainers are all experts in EU law. I also criticise those who label all Brexit voters racist, as if one needs to be racist to follow the advice of senior politicians that Brexit would help the NHS. These blanket accusations aren’t just factually wrong, they mean that when we encounter real racism, we won’t be taken seriously – when a prominent figure within the group shouted “You ain’t even **** British” at a BAME police officer, for example.

The silver lining of this mess is that we can now identify those who sow division and hatred, isolate them and make a real effort to listen to and address the concerns of ordinary Brexit voters.

• Femi Oluwole is chief spokesperson for Our Future, Our Choice