In September, 2009, just eight months into Barack Obama’s first term, when it was still possible for unsentimental observers to perceive the Tea Party’s riotous fulminations as a passing blip, Jimmy Carter remarked that opposition to the President’s agenda was driven, largely, by one thing: race. “I think an overwhelming portion of the intensely demonstrated animosity toward President Barack Obama is based on the fact that he is a black man, that he’s African-American,” Carter said. He went on:

I live in the South, and I’ve seen the South come a long way, and I’ve seen the rest of the country that shared the South’s attitude toward minority groups at that time, particularly African-Americans. And that racism inclination still exists. And I think it’s bubbled up to the surface because of the belief among many white people, not just in the South but around the country, that African-Americans are not qualified to lead this great country.

Carter, then eighty-four years old, well into the say-anything years of public life, and still mildly tainted, even among some Democrats, as a negative ideal of the Chief Executive, was quickly criticized for the remarks. Even Obama took pains to distance himself from Carter’s words. Yet, as a white Southerner (he became governor of Georgia at a time when it had barely moved past legalized segregation), a Democrat, and a former President, there was perhaps no one better suited than Carter to recognize the racial trip wires that lay in wait for the first black Commander-in-Chief.

It’s remembered now that Bill Clinton’s cultural affinity for African-Americans led to him being dubbed the “first black President,” but that’s not entirely precise. More specifically, Toni Morrison, writing in The New Yorker, presented the term in the wake of the impeachment proceedings, offering that the rhetoric around Clinton’s family background, his electorate, his personal friendships—and his appetites—had rendered him an effigy of blackness, offering a window into how the public, or a good part of it, would behave in the unlikely event that an actual black person achieved the Presidency.

That unlikely event occurred in November, 2008, and at this juncture it is difficult to disagree that Morrison was on to something. One hears it in the questioning of Obama’s American birth and legitimacy—the idea that he couldn’t really be President without trickery, that he has stolen something—and his presence in rooms where someone like him shouldn’t be. Carter’s words are, if not conventional wisdom, then certainly one of those truths that most of us know but few are willing to admit. That reticence, along with a large dose of cynicism, explains the reaction to Eric Holder’s statement, in an interview with ABC News, that the opposition to the Administration (“You know, people talking about taking their country back”) is partly driven by racism. Holder’s assessment that “I don’t think this is the thing that is a main driver, but for some there’s a racial animus” is, on the whole, more tempered than Carter’s words, and far less incendiary than Charlie Rangel’s dismissal of the Tea Party, in 2013, as “the same crackers who fought against civil rights.”

It has still been enough to inspire backlash. Senator Rob Portman, of Ohio, criticized Holder on Fox News, stating that blaming racial animus for the opposition to Obama “doesn’t help us” fight racism, adding, “I don’t think it’s a constructive statement.” The affront with which Holder’s words were greeted left one waiting for the intimation that he and, by extension, his colleagues in the Administration are “the real racists.”

The past year has offered an odd object lesson in historical redundancy. The fiftieth anniversaries of major points in the civil-rights movement tick by at the same time that Supreme Court decisions and political maneuvering in state legislatures offer reminders of what, exactly, the movement fought against. It’s fallen directly to Holder, more than to any other figure in the Administration, to respond to those echoes of history. (Jeffrey Toobin has written about his work in this area.) Holder’s office has moved against discriminatory voter-I.D. laws and police departments with a record of racial profiling, launched an investigation into the circumstances of Trayvon Martin’s death, and attempted to diminish the sentencing disparities that have bloated the number of African-Americans incarcerated during the war on drugs. It is worth noting that Holder’s appearance, last summer, at a commemoration to mark the fiftieth anniversary of the March on Washington inspired a longer and louder round of applause than that of anyone else that afternoon, save the President. Speaking to the crowd gathered on the National Mall, Holder remarked: