Who deserves to get an elite education?

That question is being debated in Massachusetts, where court papers argue over Harvard’s use of race in its “holistic” admissions process, and in New York City, where politicians are trying to increase the number of black and Latino students at top public high schools.

But the answer has always been obvious: only the elite.

While standards of merit shift over time, prominent schools and even their critics usually take for granted admissions systems that uphold the privileges of elite groups. In the United States, “elites” are mostly white people. That means Asian-Americans and underrepresented minorities — Latinos, Native Americans and African-Americans — are pitted against one another for coveted spots at elite schools.

This is patently clear in the Harvard case. A financial adviser named Edward Blum, who orchestrated Fisher v. Texas, the most recent Supreme Court case attacking affirmative action, is also behind the lawsuit against Harvard. But instead of alleging bias against whites, he and the plaintiffs use supposed anti-Asian bias as a way to undermine affirmative action for blacks and Latinos.

In doing so, however, they sidestep a more glaring inequality in admissions: Harvard applicants who are recruited athletes or children of alumni enjoy significant advantages, and these candidates are disproportionately white and well-off. However, neither the university nor Mr. Blum’s legal team address this point. In fact, Mr. Blum’s expert witness, the economist Peter Arcidiacono, excludes applicants in these “special categories” from his analysis.