Politically, this has been a good year for libertarians. Not only do they have savvy champions in the form of Sen. Rand Paul of Kentucky and Rep. Justin Amash of Michigan (among others), but revelations of unprecedented government surveillance have made Americans more skeptical of the national security state and more open to a message of restraint in foreign policy. In addition, a growing number of conservative intellectuals see a populist libertarianism as key to the future of the Republican Party, while a smaller group of progressives have floated the idea of a left-libertarian alliance on national security to push against the Washington consensus of invasiveness and intervention.

Unfortunately, if a new poll from the Public Religion Research Institute is any indication, barriers exist to both plans. If this anti-establishment libertarianism has broad appeal, you should see hints of it in the demographic make-up of self-described libertarians; the ideology should have some appeal to more than a narrow slice of the public. But it doesn’t. Of those who identify as libertarian or who have views that mark them as such, 94 percent are non-Hispanic whites, and 68 percent are men, according to the poll. As for libertarian leaners, 81 percent are white, and 53 percent are men.

Let’s start with the push to expand the GOP’s appeal through smart application of libertarian ideology. The thinking is that libertarian populism has political appeal beyond the GOP and its traditional constituencies. “Americans look at Washington and know the game is rigged against them. Conservatives can promise to level the field by getting the bureaucrats and politicians out of it,” wrote the Washington Examiner’s Tim Carney after last year’s election. “Every small businessman, ambitious immigrant, and would-be-entrepreneur should be a Republican.”

Indeed, there’s little that distinguishes libertarians from ordinary Republicans. Fifty-seven percent identify as conservative, and close to half (45 percent) say that they’re Republicans, compared to the 5 percent who identify as Democrats. Thirty-five percent say that they’re independent, but odds are good they vote Republican—if political science is clear on anything, it’s that most “independents” behave like partisans. To wit, 39 percent of libertarians say they identify with the Tea Party, which makes them less supportive than Republicans, but far more than Americans overall.

True to their ideology, the vast majority of libertarians oppose the Affordable Care Act (96 percent), a higher minimum wage, and tougher environmental regulations. All of these views place them at odds with most Americans, who aren’t as hostile toward Obamacare (44 percent support the law, and a significant percentage opposes it because it doesn’t go far enough), favor raising the minimum wage to ten dollars (71 percent), and want stronger laws and regulations to protect the environment, even if they raise prices or cost jobs.

The wide assumption is that libertarians balance their free market economic views with social permissiveness, but this is only somewhat true. On abortion, 57 percent oppose making it more difficult for women to access the procedure. Seventy percent support physician-assisted suicide, and 71 percent favor marijuana legalization. Fifty-nine percent, however, oppose same-sex marriage, and while this is significantly less than Republicans overall, it’s also a deal-breaker for most progressives, who see marriage equality as a core issue.

At a recent anti-surveillance rally in Washington D.C., former New Mexico Gov. Gary Johnson told Buzzfeed that an alliance of progressives and libertarians “should work.” And when you consider the places of intellectual agreement—from national security to criminal justice—this seems like it should be true. In practice, however, it’s hard to see how it would begin. By and large, actual libertarians are conservatives who like weed and aren’t as hostile to abortion as their more traditional counterparts. And while that’s nice, their substantial support for the Republican Party shows it isn’t as significant as it looks. When push comes to shove, their core priority is the size and scope of government’s intervention in the economy. To the rank-and-file, issues of personal autonomy and social equality—core concerns for liberals and progressives—just aren’t as important.

It’s for that reason as well that Republican elites should lower their expectations of what a libertarian turn could achieve for the party. Simply put, the people most receptive to a libertarian message—white men—are already committed to the GOP as a political institution. Women and minorities, by contrast, aren’t convinced of its relevance to their lives. And while the GOP could adopt a libertarian message on the drug war and mass incarceration—to win support from African Americans and Latinos, for instance—those groups will have to balance that against hostility to their perceived economic interests. I doubt Republicans will find themselves on the positive side of that ledger.

Libertarianism has much to contribute to our politics, and—on net—I think its influence is a welcome addition to a wide range of public policy questions. But, if this survey is any indication, libertarians are kidding themselves if they think their movement has electoral appeal to Americans. A narrow slice aside, it doesn’t.