What does a rich, privileged young man have to do to get labelled a terrorist? On 23 May, 22-year-old Elliot Rodger killed six people in California, after recording a video and publishing online a manifesto explaining that the planned massacre was an act of “revenge”. The son of a Hollywood director was taking revenge on the “sluts” who had refused to provide him with the sex he deserved. It was an act of misogynist extremism but, as soon as the news broke, commentators rushed on to social media to explain that the Isla Vista killings weren’t about sexism at all – they were the product of a “lone madman”, a side effect of a poor boy’s social impairment, and as such everyone talking about sexism ought to shut up right away.

As I write, the athlete Oscar Pistorius, who is on trial accused of murdering his girlfriend, Reeva Steenkamp, is undergoing a month of “sanity” testing to determine whether he has an anxiety disorder. If he does, this is likely to form part of his defence. In Norway, there was much debate over whether the mass murderer Anders Breivik should be prosecuted on the basis that he was insane, rather than simply “evil”. Those were the two options – Breivik was mad or he was bad. One or the other.

When wealthy, privileged or well-known individuals kill people while experiencing mental health difficulties, their psychological state is an excuse, a defence. This man was “just” crazy, so we don’t need to look at the reasons behind his actions, even if he has helpfully written a manifesto explaining those reasons (as both Breivik and Rodger did). It can’t be because he was a racist or a misogynist – it must be insanity and we should pity him.

By contrast, when immigrants and people of colour commit crimes while experiencing mental health difficulties – for example, the killers of the soldier Lee Rigby – their mental state is consistently ruled out as a factor. They are thugs, or terrorists, or both. They can’t be mentally ill – if they were, we’d have to empathise with them, so it must be their religion, or their race, turning them into vicious brutes.

Reading Elliot Rodger’s aptly titled manifesto, My Twisted World, one is struck by two things: first, that whichever way you hold up those 140 pages of sexist self-pity, you come to the conclusion that this was an extremely disturbed young man. This was a young man who had “acted out” before, who had severe social anxiety issues and whose family had clearly been concerned about his mental health.

The second thing that strikes you is that, as he grew up, Rodger’s delusions became more specific. He was obsessed with his sexual frustration and was determined to take his revenge on the women who had denied him the pleasure he felt he was owed and the men who had been favoured with their attentions. His hatred was racialised as well as gendered: he was consumed by fantasies about “pretty blondes” and wondered why “black filth” and “Asians” got to have sex with these “sluts” (he himself was half-Asian). The question is not whether Rodger had psychiatric problems. The question is why they took such a poisonous and, ultimately, tragic form.

In his book Going Postal, Mark Ames argues that the trend of office massacres and school shootings in the US is not a result of “lone madmen” but of social frustration, although both can be true. One can be lonely, furious and psychotic – and express that distress in a form that takes the existing savagery of society to its logical conclusion.

The way we talk about mental health is all wrong. For decades, the public conversation about psychosis and mental distress has resisted any analysis of social issues. If you’re depressed, or anxious, or hearing voices, it’s a chemical effect arising spontaneously in your brain – it’s the way you are. This might be considered convenient for those in power keen to overlook the structural causes of mental health problems – causes such as alienation, prejudice, poverty and isolation. That anyone can experience a mental health problem does not mean that mental illness is never political.

Mental distress is not an excuse for brutality. To suggest that does a disservice to the many millions of mental health patients who will never be violent or murderous. According to the charity Time to Change, one in three people believe that a person with a mental illness is a danger to others – but those with mental health problems are far more likely to be the victims of a violent crime than they are to commit one.

At any time, one in six people in the UK will be experiencing mental health problems and those people are chiefly a danger to themselves. I have an anxiety disorder but I have yet to attack any of my housemates or partners. The most terrifying thing I do in the middle of a panic attack is hide under a duvet watching Netflix, asking plaintively to be brought tea and perhaps a cuddle until the world stops spinning.

Yes, Elliot Rodger was mentally unwell. He was also a violent misogynist extremist. As with Anders Breivik or Lee Rigby’s killers, the question is not whether “madness” is a factor in human atrocity but why alienation, distress and psychosis take the form they do. In the case of Rodger, the sickness in our society is violent misogyny and claiming that he was “just” crazy is a convenient way to avoid looking at uncomfortable truths about the world.

Nobody is “just” crazy. To speak of mental illness and violence as if one can be either “mad or bad” is to obviate the true nature of evil, which is not the province of lone psychopaths but a product of structural oppression. Whoever we are, however we’re feeling, evil is what happens when we see other people as animals, or as things.