Ex-Tory Rory Stewart has described three East London men he met during his campaign to become leader of the Conservatives as “minor gangsters”.

The former cabinet minister, who is now standing to be mayor of London, came across the group during one of his trademark campaigning walks and filmed their conversation for his Twitter account.

He has been criticised by rivals in the mayoral race for being out of touch with Londoners over the comments, which were made during a talk in Westminster.

Stewart received both praise and ridicule on social media after the awkward exchange on Brick Lane in the summer during which he stopped the men to ask if they would make a film with him, shook their hands and said he was running to be prime minister.

One of the men said “I don’t fuck with politics man” after he told them he was a politician. The trio then quickly left while making gestures at the camera.

Stewart admitted himself “you can’t win ’em all” when he posted the message.

In an event at the Emmanuel Centre in central London on Wednesday evening he said social media had enabled him to show that politicians are listening during the leadership election.

He said: “One thing about social media is that it allows people to see politicians listening ... I can go to Brick Lane and three sort of minor gangsters can come up to me and spend a minute telling me I’m an idiot. And I can film it on my phone and put it up. And people love watching people being rude to politicians.”

Stewart’s outside bid to become prime minister ended after he was voted out of the Tory leadership race in the third round in June, after having livened up the campaign with his pop-up events, walks to meet people around the country and speech in a tent on the South Bank in London.

He was among 21 Tory rebels kicked out of the party for trying to block Britain crashing out of the EU without an agreement. He announced earlier this month that he was standing down from his seat in Penrith and the Border to run for mayor of London.

In government he served as a foreign office minister and later international development secretary.

The Liberal Democrat candidate for mayor of London, Siobhan Benita, said: “Rory Stewart’s choice of language shows yet again how he’s out of touch with Londoners. His walks around the capital are a gimmick to cover up for his lack of knowledge about the city he claims he wants to run. I’m sure Londoners will see through this stunt. Londoners need a mayor who cares, not a Tory-lite tourist who trivialises the complex issues facing the capital.”

The next mayoral election is in 2020. The current mayor of London, Sadiq Khan, is running for a second term.

A source close to Stewart said: “This was obviously a light-hearted, self-deprecatory remark – where Rory was reflecting on an amiable exchange he had when walking along Brick Lane.

“Anyone who knows anything about Rory would know it’s absurd to suggest he was implying anything malign, and if any offence was caused he’d be the first to apologise.”

In 2010, Stewart apologised to his Cumbrian constituents after saying that local people were “pretty primitive” and held up their trousers with “bits of twine”.