In August, the Justice Department sought lawyers to investigate whether Harvard University discriminates against Asian Americans in favor of black and Latino applicants. Roger Clegg, a former official under Reagan and Bush in that same department, responded by noting that “it is frequently the case that not only are whites discriminated against now, but frequently Asian-Americans are as well.”

Aside from the government’s new initiative, right now Harvard is contending with an anti-affirmative action lawsuit on behalf of an Asian-American plaintiff that is funded and organized by conservative activist Edward Blum. Blum, the man behind the gutting of the Voting Rights Act, was also the organizer behind the Fisher v. University of Texas case, which upheld affirmative action. Abigail Fisher, a white Texan, argued that she experienced racial discrimination when University of Texas, Austin rejected her application. Blum has now turned to an Asian-American plaintiff to make the case against affirmative action.

Why the sudden interest in Asian-American rights by conservatives who normally reject any mention of race or ethnicity as “identity politics,” especially when those mentions claim racial discrimination? Asian Americans are the latest vehicle for critiquing affirmative action. Blum, Clegg and others claim that providing an admissions boost to black and Latino applicants negatively affects Asian Americans.

Related: Should affirmative action be based on socioeconomic status?

The problem with this logic, however, is that it assumes that the number of seats for white students — the majority in most schools — must remain constant, while Asian Americans and black and Latino applicants vie for the remaining slots. So, under this faulty logic, giving to underrepresented minorities means taking away from Asian Americans. This slippery argument is how conservatives are co-opting Asian Americans in their mission to end affirmative action. As they do so, they assume the dominance of whites.

But affirmative action cannot explain why Asian Americans seem to need higher achievement than whites to gain entrée into top colleges. Is this a form of affirmative action for whites amid Asian-American overachievement? Most Asian Americans support affirmative action, and, in my experience, also believe that elite colleges discriminate against Asian-American applicants, in favor of whites.

A campus that is “too Asian” is seen as problematic, but one in which whites are the majority group is not.

A campus that is “too Asian” is seen as problematic, but one in which whites are the majority group is not. Why? For many Americans, whiteness is the norm, so other ethnicities must define themselves in relationship to whiteness. So, majority white campus? No problem. Majority Asian? Problem.

Affirmative action for blacks and Latinos differs from giving whites a leg up vis-à-vis Asian Americans. Affirmative action’s goal is to bring previously absent voices to campus and to address racial inequality, past and present. In contrast, colleges would be hard-pressed to make the case that Asian Americans have advantages over whites in the United States.

The United States has a history of maintaining white supremacy in college admissions, though the very definition of whiteness has shifted over time. In the past Jews were not considered part of the dominant white group. During the 1920s, elite universities worried that their campuses would become overrun by Jewish students acing admissions exams, so they changed admissions processes accordingly to dramatically reduce the number of Jewish students admitted. Here, the fear that campuses would become “too Jewish” led the way.

Eventually, of course, the anti-Semitism embedded in these concerns faded away, as Jews became part of a white American mainstream. Today, we don’t hear about the percentage of Jewish students at selective colleges, because it is no longer an issue. A burning question is whether Asian Americans will experience the same incorporation into the mainstream. I will believe it’s happening when no one balks at the possibility of more Asian Americans than white students at Harvard.

Asian Americans, as well as white, black, and Latino Americans, need to understand the history of racial exclusion and the production of racial inequality in American society. When we do, it’s hard not to support affirmative action for underrepresented racial minorities.

Still, we should not shy away from raising important questions about whether universities hold Asian Americans to higher standards than their white peers; little in U.S. history has privileged Asians over whites, so there is little unjust history to address in this case. When we disentangle the question of Asian-American discrimination from affirmative action, it’s easy to see how one could support affirmative action for black and Latino Americans yet critique a boost for whites over Asian Americans.