Attorney General Eric Holder used his Saturday commencement speech at a historically black university in Maryland to strike back at Supreme Court Chief Justice John Roberts and his argument that a colorblind society is the best way to be “faithful” to the Brown v. Board of Education decision that led to the end of the “separate but equal” doctrine.

In a contentious 2007 case involving school districts in Louisville and Seattle, a 5-4 majority on the court ruled that those districts could no longer use race to assign students to various high schools.

In his plurality opinion in the case, Roberts said the Supreme Court’s decision was “faithful to the heritage of Brown” because the “Fourteenth Amendment prevents states from according differential treatment to American children on the basis of their color or race.”

“When it comes to using race to assign children to schools, history will be heard,” he wrote. “The way to stop discrimination on the basis of race is to stop discriminating on the basis of race.

Holder did not agree, saying to graduates of Morgan State University that Chief Justice John Roberts’s argument that “the path to ending racial discrimination is to give less consideration to the issue of race altogether… presupposes that racial discrimination is at a sufficiently low ebb that it doesn’t need to be actively confronted.”

“In its most obvious forms, it might be,” Holder said while speaking on the 60th anniversary of the Brown decision. “But discrimination does not always come in the form of a hateful epithet or a Jim Crow-like statute. And so we must continue to take account of racial inequality, especially in its less obvious forms, and actively discuss ways to combat it.”

Roberts, in his 2007 opinion, noted that before the Brown decision, “schoolchildren were told where they could and could not go to school based on the color of their skin,” before emphasizing that the school districts Seattle and Louisville have “not carried the heavy burden of demonstrating that we should allow this once again – even for very different reasons.”

In his concurring opinion, Supreme Court Justice Clarence Thomas also cautioned against “elites” using race for social engineering.

“If our history has taught us anything, it has taught us to beware of elites bearing racial theories,” Thomas wrote.