Just south of Richmond, Virginia, on Interstate 95, a small group of neo-Confederates have leased a patch of land, where they’ll erect a 50-foot pole and fly the Confederate battle flag, all day, every day. On their website, the “Virginia Flaggers” say the flag will “serve to welcome visitors and commuters to Richmond”—the former home of the Confederate States of America—and “remind them of our honorable Confederate history and heritage.”

It’s hard to imagine a more self-refuting phrase. The Confederacy was a government founded on the preservation and expansion of slavery and white supremacy. “Our new government,” explained Confederate vice president Alexander Stephens, in an 1861 speech, “is founded upon exactly the opposite ideas; its foundations are laid, its cornerstone rests, upon the great truth that the negro is not equal to the white man; that slavery, subordination to the superior race, is his natural and normal condition.” There’s no question we should remember the Confederate assault on human freedom, but it’s immoral to say we should honor it.

But while shame is appropriate for the Virginia Flaggers, it’s also true that there’s nothing interesting about Confederate sympathizers—they’re a mainstay of American life. Just last month, we learned that a vocal neo-Confederate ran “new media” for Sen. Rand Paul (R–Kentucky) and ghost-wrote his first book.

What’s interesting is what the group said on Independence Day: “God bless America…and God bless those who have the courage to stand in the face of tyranny…whether it be in 1776…1861…or 2013!”

It’s hard to read this as anything other than a reference to President Obama, whose loudest opposition comes from Tea Party groups that routinely describe him as a “tyrant” who threatens American freedom. The point isn’t to compare Tea Party groups to Confederate sympathizers but is something to keep in mind when noting, as The New York Times’s Ross Douthat did earlier this week, racial polarization in the electorate. Arguing against claims that Republicans are mostly to blame for the highly racialized voting of the 2012 election, when whites pulled the lever overwhelmingly for Mitt Romney, Douthat writes that “racial bias alone can’t explain why the president went from losing non-college-educated white voters by only 18 points in 2008 to being 40 points underwater with that same demographic today.”

To blame, says Douthat, is the administration’s unabashed social liberalism and, in particular, its positions on same-sex marriage, guns, and abortion. Whereas Clinton-era Democrats tried to “win working-class whites outright,” the “Obama-era Democrats have pushed in policy directions calculated to alienate many of the swing voters who cast ballots” for conservative Democratic candidates. The New Republic’s Isaac Chotiner and New York magazine’s Jonathan Chait push back, noting the extent to which marriage, guns, and abortion aren’t racial issues, and the degree to which at least one, gun control, was spearheaded by a conservative Democrat.

But another point is worth making in support of the claim that racial bias can—or at least helps—explain Obama’s sharp decline with white voters between 2008 and 2012. Obama didn’t lose out with white voters as much as he did with white Southerners. In 2008, Obama won 49 percent of all white voters outside the South—52 percent in the East, 48 percent in the West, and 47 percent in the Midwest—compared with 30 percent of white Southerners, for total white support of 43 percent. (Disaggregating the vote further, by rural Southern whites and urban Southern whites, would likely find another large difference.)

Yes, by 2012, Obama’s white support had dropped to 39 percent, but a huge portion of his collapse came in the South, where exit polls show Obama with just 21.8 percent of white voters in the states of the former Confederacy. He saw a similarly large collapse in Appalachia, home to large numbers of lower-income whites. And while attributing the entirety of the drop to racial bias would be unfair, ignoring the role of race also seems unwise, given how closely it’s related to partisanship and the overall growth of anti-black attitudes during Obama’s first term. According to one study, the proportion of Republicans expressing anti-black attitudes had grown from 71 percent in 2008 to 79 percent in 2012.

Since we know white Southerners are heavily Republican and unusually hostile to Obama, it’s not a stretch to think they’ve experienced a large increase in anti-black prejudice as well. Or, put another way, in a region of the country where some whites continue to celebrate the Confederacy enthusiastically (see, again, the Virginia Flaggers), it’s reasonable to see a link between high opposition to the black president and racial prejudice.

That’s not to say there aren’t other grievances—as Douthat pointed out, these voters also oppose the social liberalism of the Obama administration. But looking at Obama’s dramatic decline among white Southerners, along with the history of the region, it doesn’t make sense to dismiss race as a reason for his poor standing. And indeed, it’s hard to escape the conclusion that, to some extent, Obama’s collapse with these voters was baked into the cake.