The American Bar Association wasn’t impressed with Alexander Williams Jr., and they said so. President Clinton’s first black nominee to the federal bench was rated “not qualified” by the ABA, which for most nominations, would be a death sentence. The U.S. Senate ignored them.

“We now are at a happy confluence of opinion,” then-Senator Joe Biden said in 1994 after the judiciary committee unanimously approved Williams. “We all hate the ABA.”

The American Bar Association’s influence in judicial nominations has long been a frustration for conservatives, who contend that the group is biased against conservatives. President George W. Bush stopped seeking their recommendations beforehand entirely. But liberal legal groups have long retained their own reservations about the ABA, namely that its standing committee on the federal judiciary rates women and minority candidates for the federal bench lower than white men. The committee evaluates nominees based on “integrity,” “competence,” and “temperament,” qualities which can often be in the eye of the beholder.

The Obama administration, which has nominated the most diverse slate of judges ever, has sometimes found itself struggling to confirm its nominees who have received low ratings from the ABA. The Democratic Party has more financial and political links to the legal profession, so those complaints have often only been discussed quietly – as an industry, lawyers were the biggest donors to Obama’s 2012 presidential campaign.

Now there are some numbers to back up those long whispered anecdotes about women attorneys seen as being strident or aggressive while men were simply seen as confident, or minority candidates being dismissed as lacking the qualifications of their white peers.

The potential implications for the federal judiciary are dramatic. While skin color is by no means a predictor of an individual’s legal or political views, studies have shown that have a woman or black judge on the bench can shift the outcome of cases dealing with matters like discrimination and voting rights. In 2009, according the Brookings Institution, only 20% of federal judges were women, 6% were black and 5% were Hispanic.

Maya Sen, a professor of political science at the University of Rochester, recently published a study finding that even when women and minority candidates have similar qualifications to their white male peers, the ABA committee gives them lower ratings. Sen also found little evidence that high ratings are tied to fewer reversals on the bench, raising the question of what purpose the ABA ratings actually serve in terms of predicting a nominee’s future performance. She did not find that conservatives were less likely to be rated highly.

“I tried as much as possible to compare people who were similarly situated, so I compared women candidates to male candidates across the legal education they had, work experience, legal experience,” said Sen. “What I found that even comparing across people who were very similarly situated, women candidates were 19% less likely to receive a high rating, and for African-American candidates it was 42% less likely.”

Sen’s results offer support to what groups like the National Bar Association, which was founded by black attorneys in the early 20th century, already felt was happening.

“The rating disparity issue is a longstanding complaint,” said Patricia Rosier, president of the National Bar Association. “The National Bar Association has found the same disparity.”

The ABA has disputed Sen’s findings. ABA president James R. Silkenat wrote in a letter to the New York Times that Sen dusted off Mad Men-era data” and having wrongly placed “blame for this serious problem with the American Bar Association’s nonpartisan process to evaluate the professional qualifications of potential judicial nominees.” Sen writes in her study that she compared candidates nominated by the same president in order to avoid potential generational effects.

Sen doesn’t think the discrepancy is deliberate. “The Bar Association to their credit really values diversity and they’ve put initiatives in place to promote diversity,” said Sen. “A lot people actually have implicit biases against women and minorities…as a social scientist it’s really hard to rule that out as an explanation.”

Studies have shown that racial and gender bias can persist even when unintended. A 2008 Massachusetts Institute of Technology study found that ”gender, race and nationality differences continue to affect salary growth after performance ratings are taken into account,” a 2014 study found that reviewers are more likely to review legal writing poorly if they think the writer is black. The ABA itself is cognizant of implicit bias–it has a task force whose given role is to “combat implicit bias in the justice system.”

Asked whether other factors might explain the discrepancies, Sen raised the possibility of “mismatch,” the theory that affirmative action has given underqualified minorities access to elite institutions where they’ve been unable to compete. But she said that was unlikely. “Blacks would have to perform at three times worse the rate than whites to explain away my findings,” Sen said.

Sen recommends that the ABA move to a more transparent and objective system, but also suggests that ABA ratings should not make or break a nomination as they often can. That’s an outcome the other bar associations would favor.

“I think that certainly their opinion, their rating should be considered but I don’t think it should ultimately be the deciding factor,” said Rosier.