In a buzzy new article in the New York Times Magazine, prominent libertarians make the case for why a political movement that champions the power of the individual over the collective should appeal to Americans.

The piece has some compelling moments, such as when Senator Rand Paul, the libertarian-leaning likely 2016 presidential candidate, talks about the racial inequities in the criminal justice system. “I think the war on drugs has had a disproportionate racial outcome,” Paul says at one point, echoing statements he’s made often in the past. He continues, “Three out of four people in prison are black or brown. White people do drugs too, but either they don’t get caught or they have better attorneys or they don’t live in poverty. It’s an inadvertent outcome, and we ought to do something about it.”

Paul’s support for restoration of voting rights for non-violent offenders, not to mention his recently announced support for the RLJ Rule, a version of the NFL’s “Rooney Rule” which would increase diversity in management at major corporations, makes him one of the GOP’s most prominent voices on civil rights. And yet if anyone perfectly symbolizes why minorities are unlikely to ever hop on the libertarian bandwagon, it’s Paul.

As the New York Times Magazine piece notes, Paul has been more aggressive in trying to reach out to minorities, specifically blacks and Latinos, than many of the other current GOP darlings. He has spoken before the National Urban League and at the historically black Howard University. But the reactions to his appearances highlight the challenge of combining libertarianism with the realities of American racial history and racial politics. His appearance at Howard was widely considered a dud, with even Paul himself conceding it wasn’t a rousing success.

The reason Paul struggled before a predominantly black audience of young people—despite young people being among the most enthusiastic new converts to libertarianism, according to the Times—gets at the heart of why libertarianism and minority voters will never fully click.

Paul remains haunted by remarks he made four years ago during an interview with Rachel Maddow regarding whether the federal government overreached in its implementation of the 1964 Civil Rights Act. In that interview, Paul made it a point to frequently note how he finds discrimination “abhorrent.” But, no matter how much she tried, Maddow could not get him to say the federal government was right to use its powers to desegregate lunch counters and other private businesses covered by the Civil Rights Act.

And therein lies the problem for Paul and the libertarian movement.

Following Paul’s interview, I wrote a column titled “Rand Paul is not Racist but he is Wrong.” In it I stressed (to the chagrin of some of my friends, family and colleagues), that I do not believe Paul has any animus towards black Americans. I do, however, believe he possesses the naivete that many libertarians do, namely in that they believe that limited government is a matter of principle, or efficiency—not a matter of life or death.

During a radio interview discussing my column I was challenged by a host who believed my own perception of Paul was naïve, and then challenged by a caller who was adamant about the government being intrusive, unnecessary and fundamentally wrong. Then I presented the caller with the same hypothetical I do to so many of my self-professed libertarian friends: I’m injured in a plane or car crash. There is one hospital located in the town in which the crash has taken place. Do you believe the hospital has a right to refuse to treat me on the basis of race, and that the government has no moral or legal imperative to require the hospital to treat me?

One can make a convincing argument that a florist refusing to provide flowers to a same-sex wedding, or an upscale restaurant not welcoming African Americans, aren’t really major civil rights issues. (Frankly, in this day and age, if a restaurant refused to serve me I might use the power of the Internet to help put it out of business, but I wouldn’t see the point in suing someone to serve me when there are plenty of other dining options.) But when it comes to issues like government-mandated access to health care and education for all Americans, there is more at stake.

To that end, I have yet to hear Rand Paul or any other libertarian make a convincing case for how libertarianism serves most Americans.

For the record, after a brief silence, my radio caller acknowledged that while the death of someone like me might be an unfortunate byproduct of his limited-government perspective, he still stood fully behind his philosophy on principle.

This is why as disillusioned as some African Americans, including myself, are with both Republicans and Democrats, we are unlikely to feel at home among libertarians. As long as leaving America’s most vulnerable unprotected remains a core piece of libertarianism, it is unlikely that the libertarian movement will find many allies in communities of color.