Rory Stewart declared his desire to walk through every borough of London with a spring in his step and every hope that it would bring a personal, man-of-the-people feel to his bid to become the capital’s mayor.

On Friday, it appeared to be not quite working out that way: two men who Stewart had cheerfully described as “minor gangsters” revealed themselves to be Irish musicians and called him a racist.

Speaking to the Guardian, the men – members of the rap group Hare Squead – expressed their anger at being wrongly characterised by Stewart and accused the former cabinet minister of “trying to take advantage of black boys when it’s convenient, then ridiculing them”.

The encounter took place when Stewart, then a Tory leadership candidate, strolled through Brick Lane in the summer during one of his campaigning walks and asked a group of men if they were “happy to have a quick film”.

While the men originally accepted his invitation, the chat was short-lived after Stewart revealed he was a politician. The men said they didn’t “fuck with politics” and left, calling for world peace as they did.

Stewart was criticised for being out of touch after he recalled the incident at an event this week, saying: “One thing about social media is that it allows people to see politicians listening … I can go to Brick Lane and three sort of minor gangsters can come up to me and spend a minute telling me I’m an idiot.”

Lilo Blues, one of the musicians, spoke to the Guardian shortly after finishing a driving lesson in Dublin on Friday. “It’s just another example of another weirdo white man trying to take advantage of black boys when it’s convenient, then ridiculing them,” he said.

“That guy needs to apologise. He wants to seem like the people’s man, mixing with people from diverse backgrounds, but he’s chasing clout and now he’s trying to chat shit about us. I’m not letting this guy tarnish our name, especially given the positive message we try to push in our music.

“I’m not a gangster, I’m not pretending to be and I’m not going to let some corny, goofy-looking man say I am. We’re all about peace and love, and he calls us gangsters. It was definitely racist.”

Of the encounter, he said: “I saw this opportunistic white man from a distance has noticed us three black lads and he’s, like, getting really giddy and tells his camera, ‘Lets go’ … and then there’s the awkward conversation. Then he goes, ‘OK I’m a politician,’ so I was like, ‘Nice to meet you but I’m off to get some Nandos’.”

His bandmate, Tony Konstone, whose parents moved from Egypt to Dublin when he was two, said: “The irony he’s running for mayor of London. He judged us off about 15 seconds. He didn’t even have a decent conversation with us.”

Konstone previously told the Guardian that selling drugs for a living was common where he grew up, particularly as there were lots of students, but he took another path. “We do what we do, and he’s not going to make us seem like that. To go off and make remarks after meeting us without any research is low and ignorant.”

Hare Squead have almost 1 million listeners a month on Spotify. They have been signed to a major label and toured with Nas, Joey Bada$$ and Dua Lipa. They play infectiously catchy, feel-good soulful music which fuses rap, pop, R&B, electronics and trap, and have a new single, Meeting with Myself, out on 5 November, which is about being self-aware and staying focused.

On Friday, Labour’s three most senior black female MPs accused Stewart of racism and bias.

“This is racism,” the shadow home secretary, Diane Abbott, tweeted. “And it is completely shameful. Especially when it comes from someone running to be the mayor of our diverse and multicultural capital city.”

The shadow minister for women and equalities, Dawn Butler, said his remark was an example of biased association brought about by the media’s “obsession” with “associating black men with gangs”.

“This is clear racial bias because they are black men and it perpetuates negative associations of them,” she tweeted. “Also, what the hell is a minor gangster?!”

Marsha de Cordova, the shadow minister for disabled people, said: “This baseless negative stereotyping of Londoners based on nothing other than the way they look is completely unacceptable and flies in the face of our city’s values.

“Rory Stewart should immediately withdraw these comments and apologise.”

As the row showed no signs of dying down on Friday, Stewart may reflect that his walks have been more trouble than they are worth. Sadiq Khan ridiculed them earlier this month with a reference to Stewart’s previous adventures striding across Afghanistan, saying: “I walk all the time. What I don’t do is walk around London with an iPhone filming my walk. And what I don’t do is treat Londoners like natives. We aren’t natives.”