Last week, I wrote about Marcus Aurelius, a man born into an influential Roman family who later became emperor of Rome. He is one of history’s most influential philosophers, but his family’s wealth and prestige made it easier for him to succeed than it would have been for someone less fortunate.

It was probably harder for his teacher, Epictetus, who was born a slave and is also one of the great thinkers of stoic philosophy. We don’t say Marcus Aurelius’s contribution is less, or his understanding narrower than Epictetus’s simply because he came from privilege. The ideas of both have trickled down into our moral and legal systems. We don’t hold their luck, or lack thereof, against them or their contributions.

On the basis of things I have written, and the platform I write on, my arguments are regularly dismissed while I’m called “privileged”, suggesting that points I make are less valid in some way because of my life circumstances.

Theory around privilege comes from the idea that systematic oppression makes the lives of already disadvantaged people more difficult by discriminating against them based on race, class and gender and so on, and robbing them of the right to be treated like distinct human beings.

People who finger-point privilege where they think they see it base their argument on the false premise that individualism is bad without addressing why that is the case; we are all more than the base factors others use to categorise us.

Those who try to shut down discourse by making accusations of privilege commit precisely the crime they are trying to address. They look at a person they know a few basic things about, such as where they work or what they look like, and judge them.

By that logic, people are just a statistic, a set of extrinsic attributes and not an individual with delineated personal experiences that make them different from others, whether they share a similar experience or not. The concept of privilege – as a principle – is the distasteful and dehumanising practice of lining people’s hardship up for comparison and deciding who deserves our sympathy and praise and who should sod off and count themselves lucky.

Understanding privilege – pragmatically – is something democratic governments have to wrestle with in order to fund initiatives or build programmes to assist the worst off in society and prevent systemised oppression (or ingrained wealth).

They must engage in an exercise of box-ticking based on factors such as income, gender or race, but the rest of us should take individuals on their own merit – because we can. Pointing the finger at privilege is to victimise oneself, and self-pityingly blame others for the merits that they either earned through hard work or were given through luck. The hard work should be lauded; the luck should be no more a source of judgment than it should be for those who lack it.

Most importantly, judging arguments by the people who make them is ad hominem: an argument or reaction directed against a person rather than their argument. An example would be if you said “I think a sugar tax won’t minimise child obesity”, and I responded “Well that’s because you are overweight yourself”. I’m not engaging with your point; I’m suggesting you’re not entitled to make one. This is precisely what claiming someone is privileged endeavours to do. It marginalises in an attempt to ameliorate marginalisation.

My partner is a child of immigrants, mixed-race, Jewish. He’s well-educated but from a poor background, as I am. He grew up in a Muslim area of London and went to a Catholic school. He is not considered white by white people. He is not considered black by many black people. He can only be an individual.

A point of view should be based on its own merit, not dismissed by a lazy appeal to privilege. Marcus Aurelius shouldn’t be judged negatively for the fortunate circumstances of his birth any more than Epictetus’s ideas should be praised for the abhorrent indignity of his.