Story highlights Scalia was questioning the attorney for the University of Texas, which is defending its use of race as a factor in admissions

The arguments came as colleges across the country are confronting racial tensions

Washington (CNN) Supreme Court Justice Antonin Scalia raised eyebrows on Wednesday with a comment he made during the court's hearing of an affirmative action case, in which he seemed to suggest some African-Americans belong in lesser colleges.

Photos: Supreme Court: Force for change? Photos: Supreme Court: Force for change? The Supreme Court ruled 5-4 that same-sex marriage is legal nationwide, a decision that profoundly affects the lives of millions of Americans. Some legal scholars see the court's movement on gay rights issues as proof that it is a force for change. But others say the court's role is largely the opposite. Hide Caption 1 of 7 Photos: Supreme Court: Force for change? Another major ruling upheld Obamacare subsidies; had it gone the other way, millions could have lost their health care tax credits. Some legal scholars say the court's historical mission has been to block change, not validate it, by defending the status quo and ruling in favor of "wealth, power and privilege." Hide Caption 2 of 7 Photos: Supreme Court: Force for change? They point to a 1918 ruling that struck down a federal law banning child labor, which left the practice in place for another two decades. The court said the law was "repugnant to the Constitution" because it violated states' rights. At the time, millions of children worked in dangerous mines, dank sweatshops and textile mills such as this one in Vermont in 1910. Hide Caption 3 of 7 Photos: Supreme Court: Force for change? The court has been on the wrong side of history numerous times, says author Ian Millhiser of the Center for American Progress. It issued decisions that legitimized Jim Crow segregation, approved the forced sterilization of a woman against her will and forced Japanese-American citizens into internment camps during World War II. Hide Caption 4 of 7 Photos: Supreme Court: Force for change? Other scholars such as Clark Neily of the libertarian public interest law firm Institute for Justice defend the court, saying Brown v. Board of Education -- which led to school desegregation, including in Little Rock, Arkansas, in 1957 -- shows that the court is often "better than the other branches of government and society in general." Hide Caption 5 of 7 Photos: Supreme Court: Force for change? Critics say the court makeup is part of the problem, noting that most justices have been white men from privileged backgrounds. It's an issue, they argue, that can sometimes lead to paternalistic language, as in a 2006 abortion ruling that said "some women come to regret their choice to abort the infant life they once created and sustained." Hide Caption 6 of 7 Photos: Supreme Court: Force for change? Many court critics have also questioned the wisdom of the 2010 Citizens United v. FEC decision, which opened the floodgates for campaign financing, allowing outside groups to spend record amounts. Millhiser said the ruling "gave billionaires a far-reaching right to corrupt American democracy." Hide Caption 7 of 7

Scalia was questioning the attorney for the University of Texas, which is defending its use of race as a factor in admissions in the case before the court.

"There are those who contend that it does not benefit African-Americans to get them into the University of Texas where they do not do well, as opposed to having them go to a less-advanced school, a less -- a slower-track school where they do well," Scalia said, according to the transcript. "One of the briefs pointed out that most of the black scientists in this country don't come from schools like the University of Texas."

The attorney, Gregory G. Garre, tried to interject, but Scalia continued.

Read More