For better or worse, it’s in our nature to avoid difficult conversations. And no wonder: a tough conversation poorly held can hurt more than it heals.

But avoidance only takes you so far. From my time as a youth worker I know that difficult conversations don’t become easier the longer you put them off. That’s why I know it’s time for Conservatives to have a proper conversation about Islamophobia. There is simply too much hate floating around online these days and too much of it is appearing (intentionally or not) under our banner. We must root it out, now.

Whether we like it or not, there is a subculture of Conservative supporters that place Islam at the centre of much that’s wrong with our society. And I think they’re wrong. This Tommy Robinson brigade appears to place ignorance ahead of understanding and does a disservice to British Muslims and the vast majority of Conservatives who bear absolutely no ill will toward the Muslim community. As a political movement we do ourselves no favours by tolerating or ignoring this hateful fringe.

This isn’t an argument to exclude radical Islam from scrutiny or debate. But we don’t honour that debate by indulging the unfounded fears spread by Islamophobes whipping each other into a frenzy online. It is the duty of leaders on the political right to demystify the debate, not pander to people’s cheap fear of “the other”.

Having grown up amid a strong Muslim community in Ladbroke Grove, west London, I’ve never been afraid of Islam. We were all striving toward the same goal – to succeed and make something better for ourselves. Our skin colour or religion weren’t a part of the conversation. I admired and envied the Muslim boys on the estate; most of my Moroccan mates came from strong families, something this black boy who grew up without his father wished he had experienced.

The local mosques were very receptive to my youth work when others weren’t. The local imams knew my job was to help keep their wayward boys in school and plug them into mainstream opportunities. Being in such close contact over this common cause helped us to understand each other better. We need more of that spirit of togetherness.

I’ve always known there was a problem with some of the more extreme views on the far right of the political spectrum. I’m a black man – I could hardly ignore it. When I was growing up, the BNP types on the estate used to go around looking for black boys to rough up. I was grateful to any and all who spoke out against their hatred. But watching the fallout from Christchurch and seeing the unearthing of the far-right websites and communities that helped enable it, it’s clear the problem is much deeper and broader than it was back in my day. Contrary to some powerful political voices, there aren’t “very fine people” on “both sides” of this debate.

These vicious attacks on minority communities will keep happening unless we bust the myths behind their hatred. In the Conservative party, we cannot give cover in any way to those Islamophobes, even if they vote for us. We need to crack down on hate at the source and expel it from our midst. We must set clear guidelines as to what is acceptable and then enforce them with 100% effectiveness. Full stop.

This will demonstrate solidarity with Muslim communities. And while we should also take practical measures, such as offering protection to places of worship should it be requested, our response should go much deeper. It should be Conservatives who drain the online swamp by holding tech companies’ feet to the fire. They must not be allowed to shirk their responsibility in allowing any hatred to spread.

No hateful supporters will be welcomed in my camp. This pledge is the least I can do for my Muslim brothers and sisters.

• Shaun Bailey is the Conservative London mayoral candidate