Justice Ruth Bader Ginsburg saw operas with him. Justice Elena Kagan went hunting with him. Justice Stephen Breyer traded playful barbs with him. But one liberal justice never fully succumbed to Antonin Scalia’s charm: Sonia Sotomayor. During arguments, Sotomayor often watched Scalia with a leery gaze, as though she were dreading the next trolly statement to leave his mouth. When it inevitably did, other justices wore a practiced look of long-suffering amusement. Sotomayor remained frozen in a steely stare.

On Monday, during a Q&A session at the University of Minnesota, Sotomayor confirmed that Scalia’s outré quips sometimes infuriated her. “There are things he’s said on the bench,” the baseball fan told the audience, “where if I had a baseball bat, I might have used it.”

Sotomayor added, however, that losing Scalia was like losing a family member, and that his seething dissents were simply his opportunity to “vent.” She also explained that the justices try to remember that “differences don’t stand, necessarily, on ill will. If you keep that in mind, you can resolve almost any issue, because you can find that common ground to interact with each other.”

About that baseball bat comment: I’d bet good money that Sotomayor was thinking specifically about Scalia’s comments during arguments in Fisher v. University of Texas, an affirmative action case. Scalia declared that many black students might not belong at a great school like the University of Texas, because they “come from lesser schools” and may get “pushed ahead in classes that are too fast for them.” The justice advised that black students might be better off at “a less advanced school, a slower-track school.”

An ungenerous interpretation of that comment might ask whether Scalia was endorsing a modern version of school segregation. In any event, by the time Scalia closed his mouth, Sotomayor was probably writing a scathing dissent in her head. Ultimately, she never had to put pen to paper. Scalia died two months later, tipping the balance and allowing the court to uphold the university’s affirmative action program in a fairly sweeping decision by Justice Anthony Kennedy. No baseball bat necessary.