In the oral arguments Wednesday for a Supreme Court affirmative action case, Justice Antonin Scalia—a well known critic of affirmative action—suggested that the policy was hurting minority students by sending them to schools too academically challenging for them.

Referencing an unidentified amicus brief, Scalia said that there were people who would contend that “it does not benefit African-Americans to — to get them into the University of Texas where they do not do well, as opposed to having them go to a less-advanced school, a less — a slower-track school where they do well.”

He argued that “most of the black scientists in this country don’t come from schools like the University of Texas.”

“They come from lesser schools where they do not feel that they’re — that they’re being pushed ahead in — in classes that are too — too fast for them,” Scalia said.

The case, Fisher v. University of Texas-Austin, is being brought by a white woman who was not accepted by the university and who says its policy to use race as a factor in a pool of the students it accepts is unconstitutional.

The case is unique in that the University of Texas accepts three-quarters of its students in a race-blind program (which actually was designed to increase diversity) that automatically accepts Texas high school students who are in the top 10 percent of their class. The other quarter of students are accepted through a qualitative “holistic” review that includes race along with a number of other personal and academic factors.

The challenger, Abigail Fisher, was not in the top 10 percent of her high school class and thus is challenging the holistic review program. This is the second time her case is in the Supreme Court. In 2013, the high court sent the case back to a lower court to reconsider the university’s policies under a tougher standard and in 2014, that appeals court ruled in the University of Texas’ favor.

In his comments Wednesday, it appears that Scalia was referencing this amicus brief written by UCLA law professor Richard Sander, who has also co-written a book called “Mismatch: How Affirmative Action Hurts Students It’s Intended to Help, and Why Universities Won’t Admit It.”

A decision in Fisher v. University of Texas-Austin is expected by the end of the Supreme Court’s term next summer.

Update: This story has been updated to include a fuller version of Scalia’s comments as well as more details about the case.