In 1997, Patrick Hamacher applied to study undergraduate medicine at the University of Michigan, Ann Arbor and was rejected. Hamacher, one of whose parents had studied at the university, believed he had been denied a place because he was white. At the time, the university used a points system when selecting applicants, and those from under-represented minorities automatically received extra points. Hamacher later said he agreed with the aim of diversity, but that the university's methods amount to "artificially engineering and discriminating against others – and that is just not right."

In April 2003, Hamacher took his case to the US supreme court, in an effort to challenge the principle of affirmative action. Alongside him was Jennifer Gratz, another student whose application to the University of Michigan was rejected in 1995. Gratz, whose parents never completed college, went on to apply to Notre Dame university in Indiana.

The legal challenge failed. This fall, opponents of affirmative action will have another shot with a far more conservative court. Few expect the practice to survive this time.

It's not difficult to foresee what the plaintiffs will say. Arguments against affirmative action are as predictable as they are deluded and historically selective. They'll argue that it is wrong to discriminate against people on the basis of something they cannot help – in this case, race. They'll say universities should accept the best students, regardless of their circumstance. They will say affirmative action undermines the very meritocratic ideals on which the nation was founded and has prospered, and may even go so far as to claim that it is fundamentally un-American. If they are feeling bold, they may also quote Martin Luther King Jr saying people should "judged not on the color of their skin but the content of their character".

In short, they will pitch merit against equality, as though the two were mutually exclusive, while denying a central, proven fact about how the American university system works: namely, that the central barrier to meritocracy is not race but class, and that if entrance to higher education in America were only based on test scores and academic ability, the biggest losers, by far, would be wealthy white kids.

Take Hamacher and Gratz. When Hamacher applied, the university of Michigan also awarded extra points if you were the child of alumni: an advantage he enjoyed but did not see fit to relinquish. Gratz was also turned down by Notre Dame, which gives huge preferences to children of alumni and, as a result, has a greater proportion of them than any other major university. Legacies amount to more than 20% of the freshmen class – or around twice the number of African Americans and Hispanics combined. Gratz did not file suit against legacies. Their supreme court bids were supported by, among others, then President George Bush – a C-grade student who gained entry to Yale courtesy of a legacy because his father had been there.

Along with legacies, universities also keep an eye out for applications from the children of big money donors and the politically-connected. As if the regular advantages of wealth – private education, tutors and cultural capital – were not enough, the powerful rig the system so that no open competition is possible. The scam is not just endemic, but integral to the university financing and elite expectations.

"The preferences of privilege are nonpartisan," writes Daniel Golden, author of The Price of Admission: How America's Ruling Class Buys Its Way into Elite Colleges – and Who Gets Left Outside the Gates:

"They benefit the wealthy and powerful across the political and cultural spectrum, Democrats and Republicans, supporters and opponents of affirmative action, leftwing Hollywood movie stars and rightwing tycoons, old-money dynasties and nouveau riche. They ensure each fresh generation of upper-class families – regardless of intelligence or academic qualifications – access to the premier college whose alumni hold disproportionate sway on Wall Street and in Fortune 500 companies, the media, Congress, and the judiciary."

If you were serious about looking for a single means of injecting fairness into American universities, you would target the privileged who game the system, not the under-represented and historically excluded who are trying to get a foot in the door.

"Any form of discrimination, whether it's for or against, is wrong," said Hans von Spakovsky, a legal fellow at the rightwing Heritage Foundation, who told the New York Times that his daughter was applying to college. "The idea that she might be discriminated against and not be admitted because of her race is incredible to me."

But the idea that she might benefit from discrimination on the basis of her class presumably does not bother him, even if it denies a better-qualified but less well-connected applicant.

Universities do not deny this. "We do advise the admissions office about applications coming from the children or grandchildren of significant donors," Yale president Richard Levin told the university's alumni magazine in 2004. Golden recounts how, at New York University, the associate provost for admissions, the head of fundraising and the president's chief of staff meet each Monday to discuss applications from the children of leaders in business, politics, media and entertainment.

"The list comes from different places," says the associate provost. "A dean may put us on to somebody, the board of trustees for admissions, the president, the development office, whatever. If it's a close call, the decision will go in favor of the student."

Legacies and donations aren't the only routes in for the elite. One college counsellor claimed being related to a celebrity is worth at least 100 SAT points at Brown and a few other colleges desperate to enhance their brands. Sports scholarships, widely misconstrued as a vehicle for poor black males, are disproportionately beneficial to wealthy white women in sports such as rowing, polo, equestrian events, fencing and squash. And finally, children of faculty staff get a break with admissions – and their parents are often eager to take advantage of this because they, in turn, get a considerable break with fees.

The number of those who gain college admission through affirmative action is dwarfed by those who make it through these other routes. "People always ask why black people do better in sports than they do anywhere else," Jesse Jackson once told me. "I tell them, 'When the playing field is even and the rules are public, it always gets better. But when blacks are kicking up the field and people start making up the rules … that's when the problems start.'"

The raw truth is that, given the limited places, huge over-subscription for top universities in particular, and relatively low application rates from minorities, affirmative action makes precious little difference to white students' odds. The year Gratz was turned down, minorities comprised just 11% of the applicants. Even if all of them had been rejected, the percentage of white students accepted would have risen only from 25% to 30%.

"People don't get in for a lot of reasons," said Julie Peterson, a spokesperson for the university of Michigan, when I covered the Gratz case in 2003:

"It's a myth to say, 'But for that minority student, I would have got in.' It's mathematically ridiculous. Race is a very emotional subject in our country. People have very strong feelings about it, and they are not always grounded in fact."

With the proportion of those in America who believe "most people who want to get ahead can make it if they are willing to work hard" at its lowest point since the question was first put, in 1994, it is a shame that some would target minorities as the cause of their exclusion, even as others who are far more powerful maintain their privileges. As Daniel Saracino, the assistant provost for admissions at the University of Notre Dame, told Golden, "the poor schmuck who has to get in on his own has to walk on water."

With a retired police officer for a father and a secretary for a mother, Gratz was one of those "schmucks". Those schmucks are the footsoldiers against affirmative action: too bad they cannot see that minority applicants are not their enemies but their potential allies. Too bad they cannot see that their material interests lie not in hampering the push for greater racial equality, but in challenging the entrenched practice of class inequality.

• Follow Gary on Twitter @garyyounge