John McWhorter teaches linguistics, American studies, philosophy and music history at Columbia University and is the author of "The Language Hoax: Why the World Looks the Same in Any Language." The opinions expressed in this commentary are his.

(CNN) Those who consider themselves on black people's side are having a field day dismissing Justice Antonin Scalia as a racist. His sin was suggesting that black students admitted to the most selective institutions might perform better at somewhat less selective institutions where instruction is paced more slowly.

I don't usually agree with Justice Scalia's perspectives, but we are doing him wrong on this one. Scalia didn't express himself as gracefully as he could have. No one could suppose that anything like all black students find the pedagogical pace at top-level universities overwhelming.

However, Scalia's comment stemmed not from random intuition but from research showing that a substantial number of black students would do better -- and be happier -- at schools less selective than the ones they are often admitted to via racial preferences.

The reading public's response to Scalia's point shows that few have any idea of this research or assume it was done by partisan zealots. An intelligent discussion of the Fisher v. University of Texas case now before the Supreme Court requires a quick tour of the facts.

Affirmative action's impact

First we need to clear some weeds away.

It's often thought that affirmative action at universities means admissions committees considering racial diversity only after assembling a pool of students with the same caliber of grades and test scores. Few reasonable people would have a problem with that kind of system, but it isn't what animates critics of racial preferences. Rather, the question is whether black and Latino students should be admitted on the basis of diversity with lower grades and test scores than the levels that would admit a white or Asian student.

There is no question that this has been common. It was most widely aired during the Grutter case against the University of Michigan. The question was not whether black undergraduates were regularly admitted with lower grades and scores, but whether this was appropriate. Also, the very fact of studies addressing what is called the "mismatch" between their dossiers and the schools they are admitted to demonstrates that the mismatch, itself, is real.

Now, at this point, many object that despite the mismatch, the students excel nevertheless. Here is the rub: the data is in, and in crucial ways and too often, they do not.

At Duke University, economist Peter Arcidiacono, with Esteban Aucejo and Joseph Hotz, has shown that the "mismatch" lowers the number of black scientists. Black students at a school where teaching is faster and assumes more background than they have often leave the major in frustration, but would be less likely to have done so at a school prepared to instruct them more carefully.

UCLA law professor Richard Sander conclusively showed in 2004 that "mismatched" law students are much more likely to cluster in the bottom of their classes and, especially, to fail the bar exam. Meanwhile, Sander and Stuart Taylor's book argues that the mismatch problem damages the performance of black and brown students in general.

There are scholars who dispute Sander and Taylor's thesis about undergraduate school in general. However, when it comes to the more specific points about STEM subjects and law school, takedown arguments are harder to fashion because of the simple force of the facts.

For example, on Sander's widely publicized law school paper, time has passed and few of us go in for reading law review articles. However, Emily Bazelon's widely read critique of it was hasty in claiming that the responses published along with Sander's piece refuted his claims. Rather, anyone reading them with an open mind would see that they left Sander's basic point standing tall and this applies to any other critique I have seen: there has been no "smackdown." It is similarly unlikely that anyone could tell Arcidiacono, Aucejo and Hotz that what they chronicle was mirages.

The mismatch exists in STEM and law

Quite simply, mismatch is bad news for black people in STEM and law. And if that's somehow not a problem because it isn't all of what school is, then why is it considered OK for Black Lives Matter to focus only on cop killings rather than inner city killings?

When science and the law are harder for young black people to engage than they should be, we should all call that alone a problem indeed.

And really, the fact that students thrive at different paces is hardly rocket science. It riles people coming from Scalia's mouth because it implies that black and brown students are somehow mentally inferior.

However, it needn't and doesn't. Black and Latino students are often less prepared for the pace of teaching at tippy-top schools because of the societal factors that dismay us all: quality of schooling, parents denied good education themselves, complex home lives. The question is: Do we respond to this by nonetheless placing students in schools teaching beyond what they are prepared? The data suggest this harms more than it helps, and that is not a racist observation in the least.

An example: Plato's "Republic" runs about 300 pages. At Columbia, we assign it to every sophomore as the first reading of the year. They are expected to have been able to get through it, to discuss it for two or three two-hour classes, and refer to it in a paper or two after that.

Imagine being a student who is quite bright but is from a home without many books in it. He isn't the fastest reader in the world, and his schools didn't expose him to much discussion of ideas as opposed to facts. All of a sudden, he's in a classroom where students marinated since toddlerhood in books and top-quality education are confidently discussing this book, blithely tossing off concepts he's rarely heard of, all doing a fine job of at least faking having gotten through all 300 pages.

Now imagine this student at a school where about 40 pages of the Republic is assigned, likely including the passage about the cave, with the professor making sure to usher students through the contours of the argument, aware that most of the students have rarely engaged a text of this kind. Which class is this student going to be most comfortable in, and which class is she likely to get a better grade on her paper in?

And -- given that nobody remembers much about 300 whole pages four years later, has this guy really gotten a raw deal in terms of education? Some Columbia students would be quite happy if we only assigned 40 pages and went over them with a fine-toothed comb.

What really happens

Yet the discussion of affirmative action implies that the choice is somehow between Yale or jail. But here's what happens on the ground. At the University of California, San Diego the year before racial preferences were banned in the late '90s, exactly one black student out of 3,268 freshmen made honors. A few years later after students who once would have been "mismatched" to flagship schools UC Berkeley were now admitted to schools such as UC San Diego, one in five black freshmen were making honors, the same proportion as white ones.

What civil rights leader of the past would have seen this as racism? Who in the future will? Or why are we tarring Scalia as a bigot for espousing outcomes like this in the here and now?

Our national conversation on racial preferences is underinformed and mean when founded on an assumption that anyone who seriously questions racial preferences is naive at best and a pig at worst. Affirmative action is a complex matter upon which reasonable minds will differ. With the well-being of young people of color at stake, we can't afford to pretend otherwise.