Oprah Winfrey is a successful billionaire with an empire worth $3bn, a woman whose public reputation has been built on self-empowerment. She has been frank about the stresses in her life – racism and sexism figure often – and about her struggles with her weight.

It is this last aspect that may be the hardest to deal with. Oprah's thyroid condition makes her weight problems unavoidable. She has to deal with the rebellion of her body. She may find sympathetic tailors and fabulous shoes, and accessorize brilliantly, but she likely knows what all women know: shop assistants won't be kind to women over a size 10, and that is especially true of woman of color.

To find something nice for Tina Turner's wedding, Oprah walked alone into Trois Pommes in Zurich last month, an upscale shop that carries clothes from the usual runway names – Celine, Jil Sander, Lanvin – and has locations in wealthy ski towns that attract billionaires: St Moritz, Gstaad, Basel.

A shop assistant refused to show Oprah a $42,000 crocodile handbag. Here is the incident in Oprah's words, via the International Business Times:

I was in Zurich the other day, in a store whose name I will not mention. I didn't have my eyelashes on, but I was in full Oprah Winfrey gear. I had my little Donna Karan skirt and my little sandals. But obviously, the Oprah Winfrey Show is not shown in Zurich. I go into a store and I say to the woman, 'Excuse me, may I see the bag right above your head?' and she says to me, 'No. It's too expensive.'

Oprah later mused to Larry King that she considered following the script of Pretty Woman and deploying her fortune by buying everything in the store, but decided not to give the saleswoman the satisfaction of a larger commission. The head of the luxury chain, Trudie Goetz, later said that the saleswoman didn't recognize Winfrey and that by rejecting Oprah's request she tried to be "too kind".

No doubt, the details of the incident will be pored over. It has already been attributed to racism, and rightfully so: Oprah's incident tripped a wire that worries many women of color: to be judged negatively and immediately by their race, to be treated as second-class citizens, to be pointed to the things that are not the best, but considered merely "good enough" for you. The best and most expensive, the implication goes, is saved for those with the obvious status markers: well-groomed, accompanied by a wealthy-looking man, and usually, not coincidentally, very thin.

This is what Oprah, and most other women, rarely talk about: the struggle for respect faced by women of color is shared, at times, with another group: women of size, another category to which Oprah belongs. The scale is not the same – racism can be as ugly as anything humans are capable of – yet, on a day-to-day basis, they have parallels. There is the same sense of diminishment, the same high-handed assumption by others, the same struggle for control of your own image.

Race is tied with socioeconomic status struggles; so is weight. There have been some studies of a link between a woman's size and her socioeconomic status, showing that women with low incomes tend to be of higher weight. There are many theories on this – because of unsocial working hours, lower availability of healthy food in poor neighborhoods, and other factors that may affect food choice and metabolism.

On the other side of the scale, anorexia and other eating disorders tend to be more common in people of higher socioeconomic status. You don't need science to tell you this: in the cold calculation of high-end fashion or jewellery or luxury bags, a woman's dress size is often assumed to be a marker of her status, as much as race is. Both indicate a certain institutional bigotry, an assumption based on outdated – and just plain wrong – cliches about what a person's bank balance looks like based on what their body looks like.

This is what most women know: when a woman walks into an upscale store, she has already been evaluated as to the size of her bank account or credit line (or whether she looks like she can attract a man with both of them). High-end shop assistants, like everyone who works on bonus, commission or by their wits – Wall Street traders and pool hustlers, for instance – are taught to size up a mark at a thousand paces. They read grooming, body language, clothing, and accessories as a sign of how likely someone may be to spend – or lose – money.

This is what that Zurich shop assistant was doing. Lacking the crucial information that Oprah was a celebrity, she relied on shallow markers: her size, her race. Even a Donna Karan dress is not enough to overcome those biases.

Oprah was looking to buy a handbag, which has no size measurements, but the product is not the point: in an upscale boutique, all buyers are judged, in part, by their weight. Many shops want the people in them to look "thin and cool", in the words of Abercrombie & Fitch's CEO. They may grudgingly sell larger sizes, but only online.

The key thing for people of any size is to remain hidden from areas where status is important. The rather unintelligent thinking seems to be that fat is infectious, or that thin people won't want products that have been merely glanced at by anyone over a size 10.

As a result, there is a kind of social segregation based on weight as well as on race. To test this, walk into any other fashionable shop in Zurich, or Gstaad, or London, or Paris, or Los Angeles, or New York: you may see women in flip-flops, women with messy ponytails, women in ripped jeans. Those are all perfectly acceptable – as long as she is also carrying an expensive handbag or accompanied by a man who looks like he has means. But you will rarely, if ever, see even a perfectly groomed, immaculately dressed woman above a size 10.

Shop assistants in upscale boutiques in fashionable areas of major cities have become practiced at hovering around ample women in a hurry to remind them:

Oh, I'm sorry: we don't have anything in your size.

And this size bias trickles down the economic scale: clothing retailers at all levels perpetuate the idea that carrying an extra 30lb is anti-fashion. Identifying low body weight with low status seems to infect clothing retailers from Lululemon to Abercrombie & Fitch. Last week, Lululemon said that clothes above a size 12 "are not part of its business strategy", and Abercrombie's cheerleaders-only aesthetic has become the stuff of legend. "A lot of people don't belong (in our clothes), and they can't belong," says Abercrombie CEO Mike Jeffries.

This is, of course, ridiculous. There are many reasons for a woman's weight to vary, including everything from medication to thyroid issues to stress to lifestyle. Fat-shaming, as the fashion retail industry often indulges in, is also a form of fat-blaming: it assumes that a woman's weight tells you all you need to know about her. This is irrational and, most importantly, completely false.

The common bias of tying socioeconomic status with weight is just bad business. The saleswoman who made that assumption about Oprah was wrong – and her shop not only lost a high-value customer, but is also paying a steep price in the very public wrath of a billionaire scorned. Whatever the social formula those shop assistants are using may be, it often fails to truly judge a customer's ability to spend. It was, to quote Pretty Woman, "a huge mistake".

There are women who would like to have access to upscale fashion and have the means to indulge – if only shops would recognize that high income and good credit comes in all shapes and sizes. Maybe clothing retailers should, similarly, learn that a woman's cultural, economic and social power is not solely discernible at a glance – either by her skin color or by her weight.

It might help their bottom line if they stopped judging those of others.