Sen. Rand Paul of Kentucky has been leading an earnest—if sometimes awkward—effort to court black voters in the wake of President Obama’s re-election. It was a stretch to begin with, but Paul’s outreach campaign crashed into a wall on Tuesday as news emerged that a staffer is neck-deep in neo-Confederate ideology.

Jack Hunter joined Paul’s staff in 2012 as a social media aide and previously worked on one of Paul’s books. But as the conservative Free Beacon reports, he’s also a former shock jock who went by the moniker “the Southern Avenger” and wore a Confederate flag mask to events. More disturbingly, he served as chairman at a secessionist neo-Confederate organization, has a major crush on John Wilkes Booth, and a strong interest in protecting white culture. For Paul, who has been working hard to position himself as a legitimate mainstream option ahead of an expected presidential bid, Hunter’s presence is a major setback.

Hunter told the Beacon he’s changed his views, although he still believes the Civil War shouldn’t have been fought. A spokeswoman for Paul told them that “Sen. Paul holds his staff to a standard that includes treating every individual with equal protection and respect, without exception.”

But Hunter’s background highlights a major obstacle for Paul as he tries to market himself to a broader audience than his father, former congressman and presidential candidate Ron Paul. The states’ rights libertarian scene that the father and son duo emerged from is inextricably tied to neo-Confederate and extremist culture. It’s not a coincidence that Hunter moved up the ranks any more than it’s a coincidence that the elder Paul’s new think tank is loaded with similar fringe figures or that the former congressman published a racially inflammatory newsletter for years. It’s not a coincidence that the elder Paul also thinks Lincoln was a tyrant who never should have stopped the South from seceding. Rand Paul has kept his distance from Paul’s foundation and isn’t his old man, but he’s also long been a strong supporter of his dad’s political career and relied heavily on the movement his father built to launch his own in Kentucky.

Sen. Paul has seemed genuinely confused as to why these kinds of ties are so problematic. At Howard University, he argued that black voters should give Republicans another look because Abraham Lincoln was a Republican. But he refused to acknowledge the idea that his own base’s more recent history with race might be preventing them from taking his words seriously. And Paul, who famously attacked the Civil Rights Act as an intrusion on private property while running for Senate, complained at Howard he was misunderstood even as he repeated his criticisms of the anti-segregation law in the very next sentence.

Paul’s libertarian and anti-interventionist philosophy has found new relevance in recent years as many voters grow concerned by creeping threats to civil liberties or a foreign policy they perceive as needlessly aggressive. But until he grapples with the post-Civil War history of not just the Republican Party, but his own narrow libertarian strain and its unfortunate ties to blatantly racist movements, he’s going to be hampered in his efforts to attract a mainstream following.