A new series of Would I Lie to You?, a TV panel show on which I'm a team captain, has just been recorded. The other captain, Lee Mack, is similar to me in many ways: we're both comedians; we're about the same age; we write as well as perform; we have both acted in sketch shows and sitcoms as well as appearing in panel games; we're both team captains on Would I Lie to You?. But, for the comic purposes of the show, our differences are emphasised. Since Lee comes from Southport and has a Lancastrian accent and I'm a southerner who sounds vaguely RP, he's characterised as a woodbine-smoking Jarrow marcher whose whippet died of rickets and I'm Bertie Wooster but with a less burning sense of social injustice. We eat only tripe and swan respectively.

Illustration by David Foldvari.

Comic stereotyping is handy on shows like that. It means other panellists can tap into the audience's cultural assumptions about being "northern" or "posh", which is easier than searching for shared assumptions about our inconveniently specific personalities, and allows us either to play up to the stereotypes, to subvert them or to pretend to be irritated when Rob Brydon makes reference to them, before countering with a remark based on lazy stereotyping of the Welsh. (I'm half Welsh myself so I lose a little part of my soul every time I do that, but then getting a laugh can fill the most cavernous vacuums in the soul.)

This is all fine as a joke-generating technique. Everyone's taking it as well as dishing it out and, because our primary aim is the shared one of making a comedy show, we can emphasise differences without being divisive. Problems only arise when people start to take stereotyping seriously: many a true word may be spoken in jest but there's also a lot of exaggeration and nonsense.

The government suffers from this. There are several men in the cabinet who've inherited money and been sent to expensive schools: toffs basically. That provides comic opportunities to take the piss out of them, and the handful of true words spoken in those many jests imply that ministers' wealthy backgrounds might make them act in the interests of money and privilege, not those of the people who voted for them, let alone the broader electorate. And that their privilege-skewed life experiences mean they won't know what Britain is really like if you're poor, so that, even if they have compassionate instincts, they will never have been confronted by the injustices to provoke that compassion. So we joke about little Lord Fauntleroy sipping champagne in a cash-lined peasant-skin yacht, squeaking: "No more benefits!"

But it's wrong to infer from the jokes that being born into a rich family means you're a bad person or that judging politicians on their backgrounds rather than their actions is fair. That attitude leads to behaviour such as that exhibited by Labour in the Crewe and Nantwich byelection campaign of 2008. Labour activists targeted the wealth of the Tory candidate rather than anything he said, mocked him by dressing up in top hats and used the slogan: "Do you want a Tory con man or a Dunwoody?", referring to the fact that their candidate was the daughter of the late Gwyneth Dunwoody, the previous MP.

The hypocrisy of this call to heredity over riches is breathtaking. The justification for suspicion of the posh is the fear that their money or connections may give them unwarranted advantages. But that's not an accusation you're on safe ground making if you're the last MP's daughter.

I reckon it's OK to be snide about Tory poshness while complaining about the cuts, as long as you're clear that it's the cuts you think are unacceptable, not the poshness. But I'm slightly worried that some people are forgetting that distinction and, as someone who's often been called posh, that makes me nervous. So does a tweet I was sent last Saturday, the day of both the cuts protests and the Boat Race: "you and the Tristrams off to watch plummy cunts row today? [I wasn't.] A day for those with the means to help themselves."

I know it's unwise to take any individual remark from the internet seriously but, from what I can tell, this tweeter isn't alone in lazily associating the Boat Race, and indeed anything to do with Oxbridge, with unaccountable privilege. As someone who went to Cambridge, this makes me feel sad – and a bit guilty that, for ease of comedy, I've allowed myself to be stereotyped as posher than I am, in a way that might allow people to assume that Cambridge is more of a bastion of privilege than it is.

I'm not denying that the average Oxbridge student is higher up the class system than the average citizen, but that doesn't make it right to characterise those universities as higher education equivalents of Eton and Harrow when they're state-funded institutions. They used to be free to anyone who achieved the required grades and, even now, they'll cost no more than many of their less eminent rivals. Publicly referring to their annual rowing race as if it's an aristocrats-only water fight only worsens the regrettable, and for many years diminishing, class imbalance within them. It will put off applicants who would both benefit from and benefit Oxbridge.

Our society is deeply divided. Occupying Fortnum & Mason, a luxury food shop flourishing despite austerity, is a neat way of illustrating that. Pointing out that some people chose to watch a student sporting fixture instead of protesting is not. Anyway, as a Tory newspaper noted with consternation, at least one of the Fortnum occupants, Adam Ramsay, is incredibly posh himself: his family have a castle. The paper implied that, because of his birth, his opposition to government cuts was hypocritical – after all, everything's all right for him. In response he said: "I think people with all kinds of backgrounds are starting to see that these cuts threaten to undermine our economy and ruin people's lives."

That's the key. People's backgrounds, or our inaccurate assumptions about them, mustn't be used as a reason to ignore what they say – to dismiss them with: "Well, you would say that!" By disregarding people's views as merely a product of their upbringing, you also absolve them of responsibility for them. You make George Osborne's rich family excuse his policies – he doesn't know any better, so he would do that. Well, it's not an excuse, even if it might be a reason. And speculating about that is much less important than listening to his arguments and explaining why they're wrong.