To the conservative Heritage Foundation, comprehensive immigration reform is an epic boondoggle. To wit, in a report released earlier this week, Heritage puts the cost of immigration reform at a whopping $6.3 trillion. That’s nearly half the size of the United States economy.

But there’s a problem. To come to this number, Heritage assumes that unauthorized immigrants will claim the full array of federal benefits as soon as they become citizens. As Heritage president Jim DeMint explained on ABC News’ This Week, “We just want Congress, for once, to count the cost of a bill. They’re notorious for underestimating the cost and not understanding the consequences.”

Not only does Heritage assume a world where every unauthorized immigrant becomes a citizen, but it assumes one where upward mobility has disappeared—every immigrant is taking more in benefits than paying in taxes—and one where there are no economic gains from legalizing and integrating immigrants.

In the push to understand why Heritage would make such assumptions, Dylan Matthews of The Washington Post discovered an important fact about one of the coauthors, Jason Richwine, a “senior policy analyst” at Heritage. Richwine earned his Ph.D. in public policy from Harvard University in 2009, with a dissertation titled “IQ and Immigration Policy.”

His thesis is straightforward and clearly stated in the abstract: “The statistical construct known as IQ can reliably estimate general mental ability, or intelligence. The average IQ of immigrants in the United States is substantially lower than that of the white native population, and the difference is likely to persist over several generations.”

He then goes on to prove each point. On the question of IQ, he relies on a broad variety of research, from the American Psychological Association—to underscore the extent to which established measurements have found racial differences in intelligence—to the work of the late J. Philippe Rushton who argued that these differences were genetic in origin. He cites one article that deals with the differences of the “negroid brain.”

In trying to explain the IQ differential between races, Richwine borrows from Rushton, arguing that “the totality of the evidence suggests a genetic component to group differences in IQ.” He tries to hedge against accusations of racism by noting that you can’t blame “obvious biases in test construction and administration” for the differential. And he gives an example of what he means when he says one racial group is less intelligent than another. These differentials, he writes “places the average black at roughly the 16th percentile of the white IQ distribution.”

There are a host of problems with “IQ” as a measurement, including the degree to which it corresponds with what we understand as “intelligence” and the extent to which IQ measurements reflect long-term systemic bias. Let’s set those aside. The more obvious problem is with trying to measure “racial” differentials in the first place.

Remember, racial groups are imagined communities; there’s nothing biological or genetic that makes someone “black” or “white.” These are social distinctions. What does it mean to measure “Hispanic” intelligence, when Hispanics come from a huge variety of socio-cultural backgrounds and environments? Hispanic isn’t a “population” as much as it is an umbrella term for populations, namely, people with heritage in the Spanish-dominant countries of the Americas.

The same is true of African-Americans or black people; there is no such thing as a “black IQ” that you can compare to other “blacks” around the world. Rather, there are the IQs of different peoples of African descent, which you can try to cross-compare, accounting for vastly different social, cultural, and economic circumstances.

It’s for this reason and many others that people are skeptical when anyone attributes persistent racial difference to genetics. It doesn’t help that proponents of the view tend to overlap with the fringes of Western life. Rushton, for example, headed the Pioneer Fund, listed as a white supremacist organization by the Southern Poverty Law Center for its funding of openly racist groups and support for “research” into ideas of racial inferiority for blacks and other groups.

In his dissertation, Richwine writes that “No one knows whether Hispanics will ever reach IQ parity with whites, but the prediction that new Hispanic immigrants will have low-IQ children and grandchildren is difficult to argue against.” More broadly, Richwine argues that the relative IQs of immigrant groups should be weighed when determining who receives permanent entry into the United States.

Not only is it easy to see the similarities between Richwine’s ideas and the “scientific racism” of groups like the Pioneer Fund, but those ideas also mesh with the policy conclusions of the Heritage report, which calls for a greater emphasis on high-skill immigration, as opposed to bringing low-skill migrants.

Heritage has distanced itself from Richwine and his dissertation. “This is not a work product of The Heritage Foundation. Its findings in no way reflect the positions of The Heritage Foundation,” says Mike Gonzalez, vice president for communications at the organization. But the entire situation raises questions.

Last year, National Review fired two writers, John Derbyshire and Robert Weissberg, over racist rhetoric, and in the case of Weissberg, ties to a white supremacist organization. But that rhetoric wasn’t as unusual as you might image. It’s not at all hard to find examples of conservative figures—politicians, activists, writers—who have had to apologize for racist remarks. Last year, for example, the president of the North Carolina-based Locke Foundation had to apologize for posting a graphic of President Obama eating fried chicken.

The Republican Party isn’t a racist organization, and conservatism isn’t a racist ideology. Nonetheless, over the last four years, the figures and organizations that have hit hot water over racially charged rhetoric have been overwhelmingly right wing. And survey after survey shows a conservative community that’s more likely than other Americans to hold negative opinions about blacks.

It’s hard to pinpoint what exactly is going on, but there’s more than enough information to conclude that if the right wing is attracting racists and white supremacists, it might want to reevaluate its approach to politics.