On Monday, Extinction Rebellion co-founder Roger Hallam put his foot in it. Completing a six-week stint on remand after being charged with flying a drone near Heathrow Airport, the 53-year-old told supporters that his time behind bars had been "pretty much as good as it gets".

Famously, prison isn't always so kind to its residents, and critics pointed out that Hallam likely had an easier time inside because of his race and class. The race element is easy to accept – the vast majority of prison guards in the UK are white, and far-right gangs now dominate wings in some jails, so it stands to reason that a white inmate would have a comparatively easier time than a BAME inmate. But the question of class is less clear cut.

The general perception of prison is that anybody whose face doesn't fit is likely to be in for a hard time. Evidence shows that working class people are disproportionately represented inside – partly because wealthier people who have committed crimes are likelier to get off with a more lenient punishment – meaning, in prison, few faces fit less than those belonging to men named Rupert.

But is this perception rooted in reality, or does social standing have the same insulating effect behind bars as it does outside?

Reformed drug smuggler David McMillan grew up as the son of a wealthy TV company manager, but became embroiled in the underworld after meeting associates of thieves and safecrackers while working at a seedy porn-theatre in his early twenties. He's served numerous prison sentences, most recently in 2012, at HMP Rochester, for smuggling heroin.

David disagrees with the notion that prisoners from backgrounds like his have it easier, telling me middle class convicts are frequently met with scorn. "Being middle class in prison doesn't help at all," he says. "Well, not if you show any expression of a class distinction. It only generates resentment, with people thinking, 'You don't have the credibility to be here; you broke the law on a whim, not out of honest working class need.'"

McMillan does however concede that certain middle class mannerisms can be used to an inmate's advantage: "The faux politeness of the civilised folk usefully makes the 'scum' with whom one is locked up believe you actually care about their problems – that's not my view, but certainly that of the bankers I met in London's big Wandsworth prison," he tells me.

Like McMillan, William Garnier enjoyed a privileged upbringing, but later became involved in serious organised crime, leading to a six-year sentence for transporting cocaine. He says if inmates become aware that a prisoner is from a wealthy background, they can be targeted for bullying. He also says that middle class cons are often stereotyped as paedophiles because they don't fit the bill of a stereotypical criminal. "A lot of people thought I was in for sex crimes against kids," he tells me.

Given this stereotype, I wonder if the middle classes fare any better on the vulnerable prisoner unit (VPU), which is reserved for inmates who are at risk of being attacked in the mainstream prison population. This wing typically contains criminals from a broader range of social backgrounds.

"If you let people know you're not one to be bullied then you're not more at risk of bullying or victimisation," says former inmate Joel, who spent time on a VPU. "I definitely wasn't victimised for being middle class."

Joel adds that although he doesn't think his background made a whole lot of difference to his quality of life during his sentence, the fact he had access to money made it slightly easier. He was a finance professional prior to being jailed, and also had family members with some cash at their disposal to put on his books. "My parents used to send me £60 a month towards canteen [items available from the prison shop] and phone calls," he says. "I worked as well, so I always got good money for the canteen. I managed to get a duvet and pillow and a digital radio."

Given that two out of the three ex cons say being middle class actually makes time inside more difficult, with one believing it doesn't make much difference, why did Hallam appear to have such an easy ride?

According to former prison guard Neil Samworth, it's more likely to do with the length of his time inside, the fact he was on remand and the strength of his support network.