Justice Antonin Scalia thought it shouldn’t make too much difference who sat on the bench. If Supreme Court justices hewed to his originalist philosophy, deciding cases was a matter of applying objective criteria — personal experience shouldn’t enter into it.

But in his dissent to Obergefell v. Hodges, which found that the Constitution guaranteed a right to marriage for same-sex couples, Scalia argued his colleagues had every reason to be concerned about diversity. A group of nine people who look so little like America couldn’t be qualified to do the democratic work of making law. And making law (the proper function of the legislative branch, not the judiciary) was exactly what he thought his colleagues were trying to do.

Scalia ticked off the ways he and his colleagues fell short of being representative of the country:

Take, for example, this Court, which consists of only nine men and women, all of them successful lawyers who studied at Harvard or Yale Law School. Four of the nine are natives of New York City. Eight of them grew up in east- and west-coast States. Only one hails from the vast expanse in-between. Not a single Southwesterner or even, to tell the truth, a genuine Westerner (California does not count). Not a single evangelical Christian (a group that comprises about one quarter of Americans), or even a Protestant of any denomination.

How likely is the next nominee or confirmed justice to address the problem of diversity and representation on the Supreme Court? We don’t know the full pool of candidates President Obama is considering, but we do know one especially likely source of nominees: circuit courts. Seven of the eight current justices were serving on circuit courts when nominated, and so are several of the potential Scalia replacements whose names are being floated.

After paging through all 158 of the active federal appellate judges’ official biographies, articles about their appointments, and, yes, their wedding announcements, I found that the circuit courts are more diverse than the Supreme Court. That is, they’re more diverse by two of Scalia’s metrics: where the judges grew up and where they went to school.

Just over a quarter of circuit court judges attended either Harvard or Yale Law School, offering a number of options for breaking those schools’ lock on the Supreme Court. But there may be a political factor that favors the nomination of those schools’ alumni. Linda Greenhouse, who has covered the Supreme Court for The New York Times and teaches at Yale Law School, said, “Presidents want sitting judges with impeccable credentials, and these two schools count.” In other words, presidents want their nominees to get through the often contentious nomination process, and it may be an easier sell if the nominee went to a brand-name law school.

Compared to educational diversity, geographic diversity is harder to find. More than half of circuit court judges (54 percent) hail from the East or West Coast.

Looking for judges who satisfy both categories narrows the pool considerably. Only 54 circuit court judges neither attended Harvard nor Yale Law School nor grew up on a coast. But not all of those people are plausible Obama nominees. When I ignore those who were not nominated to their current posts by other Democratic presidents, I’m left with a pool of 29 judges.

Even that includes some people Obama will probably never consider. Obama wouldn’t be likely to pick the two judges in their 80s who match my criteria, and if he’s looking for someone under 60, he only has 11 circuit court judges to pick from. As the pool gets shallower and shallower, you can see how difficult it can be to shake up the court’s demographics.

And I still haven’t touched Scalia’s last criterion: Finding any kind of Protestant to add to the exclusively Jewish and Catholic court. But data on the religion of sitting judges is difficult to come by. I called two of the 12 circuit courts, and both their court officers declined to comment on judges’ faith.

Even if Scalia intended his indictment of the homogenous court primarily to ridicule his colleagues, Jonathan Kastellec, an assistant professor at Princeton University, told me that changing the composition of the court really could change its decisions. In his own research, he’s found that not only do African-American judges differ in how they rule compared to judges of other races, but also that their presence makes a difference. Their colleagues vote differently when serving in a racially mixed group compared to an all-white one. Other researchers have observed similar effects on male judges who serve with women, rather than with all men.

If adding judges from a broader range of faiths, schools and hometowns did shift the court’s decisions, Scalia wouldn’t have seen it as a victory for diversity, though. He would have viewed it as damning evidence that his colleagues were deciding cases on something other than what the Constitution’s creators intended. If he could influence the nomination process at all, he’d probably repeat what he’d said before: If he couldn’t have someone who agreed with him, he just wanted someone smart.