[Epistemic status: As always, I am not a statistician, and anything I say should be taken with a grain of salt until confirmed by others]

I.

A while ago, I wrote Beware Summary Statistics, where I talked about all the ways I’ve been misled by things like r-values and so on. I recently found some really interesting cases that brought up a few more some of these issues.

Back in June, Noah Smith blogged about a study on IQ And The Wealth Of States.

Some background: a group including Richard Lynn suggests that IQ is the driving factor behind income differences among countries. They are able to cite statistics on how a very rich country like Singapore has an average IQ of almost 110, and a very poor country like Haiti – well, it’s hard to say, because not too many Haitians take IQ tests and the ones who do might be so confused by this weird new idea of filling out a written multiple choice test that they choke and underperform – but officially Haiti has an IQ of like 70. Since you need smart people to build cool things like highways and power plants, maybe this explains a lot of the development/underdevelopment dichotomy. These people can point to a pretty good correlation between national IQ and national development to support their thesis, but the obvious counterargument is that maybe highly developed nations have good health and good education which raises IQ.

Anyway, the study Noah blogged about tested the application of this theory to US states. Noah sums up the results as follows:

The upper bound for the amount of state income differences that can be explained by population I.Q. differences is about a third. If we assume that achievement scores are a good measure of I.Q. and that school attainment doesn’t improve I.Q. very much, then the number goes down to about one-sixth. What this really shows is that there is Something Else that is driving state income differences. My personal guess is that this Something Else is mainly “external multipliers” from trade (the Krugman/Fujita theory). Institutions probably play a substantial role as well (the Acemoglu/Robinson theory). That’s certainly relevant for the debate about different models of capitalism, where we often compare the U.S. to Scandinavia and other rich places. In any case, this result should be sobering for proponents of I.Q. as the Grand Unified Theory of economic development. Average I.Q. is not unimportant for rich countries, and we should definitely try to raise it through better nutrition, education, and (eventually) brain-boosting technologies. And it still might matter a lot for some poor countries. But for rich countries, there are things that matter a lot more.

II.

Let’s go to another study. The Atlantic has an article on How Rich People Raise Rich Kids, which is about Black, Devereux, Lundborg, and Majlesi (2015). They look at adopted Swedish kids and determine whether their wealth (not income!) as adults is more correlated with that of their biological parents or their adoptive parents. They find that non-adopted kids’ wealth correlates with that of their non-adopted parents at 0.33, adopted kids with biological parents at 0.13, and adopted kids with adoptive parents at 0.23. This suggests that upbringing is more important than genetics in determining how much wealth you will have.

Part of me wonders if an adoption study is really the best way to deal with this. Giving your children up for adoption is a very unusual choice, which means the biological parents are a very nonrepresentative group – and the study indeed finds that even forty years later, these biological parents have only a third as much money as the average Swede. If the same factors that cause them to give their children up for adoption – illness, relationship problems, trouble with the law – also cause them to fail to live up to their “genetic potential”, then we wouldn’t expect their children (who may lack these issues) to be correlated with them. The extremely odd shape of the graph also gives me pause: after a certain point, the wealthier your biological parents were, the less likely you are to be wealthy. Why? Certainly there’s no such effect for adoptive parents or non-adopted people!

But nitpicks aside, I am pretty willing to believe this. Although other studies have found evidence that biology is more important than upbringing in determining income (not wealth!), wealth seems like a different story. For one thing, you can just give your kids money! As I said last time we talked about GiveDirectly, there is pretty good evidence that giving people money causes them to in fact have the money which you just gave them. The current study reasonably tries to avoid having to deal with inheritances by looking at people whose parents are still alive, but even living parents can give lots of money to their children (for example, I come from a pretty wealthy family and my parents gave me lots of money, which I mostly used to help get through medical school without much debt. This means right now I have more “wealth” than people who took out bigger loans).

The authors write that:

While we have established the relative role of nature versus nurture, the exact mechanisms of wealth transmission are more deifficult to ascertain. Wealthier parents tend to be better educated and earn higher incomes, and these factors could lead to the increased wealth of their children through, for example, teaching them about investment opportunities or providing the right opportunities. However, when we investigate this, we find little evidence that this is the case. It may also be that wealthy parents invest more in their child’s education and career, which could then lead to higher child wealth accumulation. When we examine whether this is the case, however, we find little evidence for education or income as mechanisms. So the pathway through which parental wealth affects child wealth does not appear to be primarily parental schooling and income or child human capital accumulation and greater labor earnings. Taken together, our findings suggest potential roles for intergenerational transmission of preferences (children of wealthier parents may choose to save more or invest in assets that have higher returns) or for financial gifts from parents to children. Unfortunately, we do not have information on savings behavior or on financial gifts so this evidence is only suggestive.

So it seems to be a matter of how much money your parents give you, rather than of you learning deeply important personality traits from them or something. Fair enough.

But I got distracted. I was talking about the Atlantic’s article about the study. What did they have to say?

Even when they’re adopted, the children of the wealthy grow up to be just as well-off as their parents. Lately, it seems that every new study about social mobility further corrodes the story Americans tell themselves about meritocracy; each one provides more evidence that comfortable lives are reserved for the winners of what sociologists call the birth lottery…What appears to matter—a lot—is environment, and that’s something that can be controlled.

III.

Let’s talk about three things – correlation, percent variance explained, and reality.

(I’m talking a big talk here, but I only got a good feeling for this when I asked various people on Tumblr to explain it to me. But they did a good job, and now I’m explaining it to you.)

Correlation is an r value. Percent variance explained is correlation squared. Reality is best viewed in the form of a graph.

Noah tells us that the IQ-of-states study found that only about 14% of the variation in state GDP was explained by IQ. Since variance = correlation^2, this implies that there’s a correlation of sqrt(0.14) = 0.37 between state IQ and state GDP. The paper itself did some sort of super high-powered nuclear statistics to arrive at this estimate, but I took lists of state average aptitude test scores and state GDP per capita and correlated them together in SPSS and got 0.40, so easy way and hard way agree pretty closely.

Here’s the graph associated with the study (I added the line):

(Proposed new state motto: “Louisiana – Where We Succeed Wildly Out Of Proportion To Our Low Intelligence!”)

Huh! When you hear “…only explained 14% of the variance” it sounds like “go home, this is boring,” but when you hear “correlation of 0.37”, it sounds like “huh, they seem pretty related”, and when you see the graph, it looks like “holy frick, everything is IQ after all”. But all of these are the same finding!

Now. Consider the Swedish study and the Atlantic article about it. They say that although biological parents were correlated at r = 0.13, adoptive parents were correlated at r = 0.23. Therefore, they conclude, nurture wins over biology, meritocracy is a myth, everything depends on the lottery of birth, and wealthy parents are foredoomed to have wealthy children.

But r = 0.23 means the percent of variance explained is 0.23^2 = ~5%. If some Social Darwinist organization were to announce that they had evidence that who your parents were only determined 5% of the variance in wealth, it would sound like such overblown strong evidence for pure meritocracy that everyone would assume they were making it up.

The study didn’t come with a scatter plot, but here’s a plot from a totally different study that got a very similar correlation (0.24) to give you a feel for what it might look like:

The article makes it sound like your position in the birth lottery determines your destiny with impressive finality. The correlation seems unimpressive. The variance seems really unimpressive. The scatter plot looks like someone took random noise and drew a line through it. Once again, all the same finding.

Which of these three ways of presenting the data is most accurate? Um. Hard to say. I asked some people whether correlation (ie r = 0.23) or variance (ie 5%) is a better description of how the world actually works. That is, given that I have a certain “feel” for how much people differ in wealth, and a certain “feel” for what it means to win the birth lottery by getting rich parents, should I feel like the birth lottery thing explains 23% of how much wealth you have, or only 5%?

(I was only a Discordian for like six months, in my freshman year of college, but I still end up with fives and twenty-threes every time I try to do something involving numbers)

The answers I got were that it’s complicated, and both sort of work even though intuitively they should be mutually contradictory. The distribution of wealth is consistent with a story where it is explained by twenty different factors, each of which is just as important as parental wealth, which is sort of like 5%. But parental wealth explains just over a fifth of the standard deviation in wealth, which is sort of like 23%. The best explanation I got, from an anonymous commenter, was this:

About variance: consider the following. Flip 25 coins. Each heads gives you +1 utility point, and each tails gives you -1 utility point. One of these coins is labeled “upbringing”. On average you get 0 utility points. But you can also expect not to get exactly 0: on average, your distance from 0 will be 5 (the stdev is 5). So this is a little similar to a single coinflip that gives you either +5 or -5. Changing your upbringing from -1 to 1 gives you 2 points, out of a typical range of -5 to 5.

There was also a general consensus that if I had to think about this intuitively, which I should try not to do, 5% was probably the number that would lead me less astray, at least in terms of inputs. So fine. Whatever. Five percent it is.

IV.

Stalin once said that “The people who cast the votes decide nothing. The people who count the votes decide everything.”

(I briefly questioned whether Stalin really said that – like, I know he was an evil despot, but I’m not sure he was sufficiently self-aware about being an evil despot to come up with witty evil-despotism-related quotes. But I checked his WikiQuote page, and not only is the saying well-attested, but it seems Stalin was totally all about coming up with the witty self-aware evil-despotism-related quotes. Huh.)

In the same way, the people who conduct a study decide nothing. The people who report on the study decide everything.

I think Noah and the Atlantic were both honest and did a decent job reporting on their individual studies. But taken together, Noah concluded “This shows that IQ doesn’t really matter that much in explaining GDP” and the Atlantic concluded “This shows that who your parents are matters a colossally huge amount in explaining wealth” when in fact if you put both the studies side-by-side the IQ finding is three times as strong as the parents finding.

[EDIT: Some people have been misunderstanding this, so let me say it clearly. These are two studies about two different things! It’s like if I said the percent of weight gain explained by carbohydrates is three times as large as the percent of crime explained by poverty. I can compare these two things statistically, but I’m not trying to combine them into a single meta-study where I say that carbohydrates cause more crime than poverty! Also, some people seem to think I’m saying the Swedish study finds genes/biology/IQ to be more important than nurture. It doesn’t – in fact, it finds the opposite! Nurture is more important than genes but in the grand scheme of things both are tiny and the variance is almost entirely due to other things or randomness.]

In the end, nobody except a handful of researchers is going to remember the exact number. But they might remember “There were a couple interesting studies recently, one of them proved state IQ didn’t matter, the other proved that who your parents are totally determines whether you get ahead in life.” Framed that way, you might actually have gained negative knowledge from your diligent attempt to understand the economic literature.

And if the surrounding culture is pretty united in wanting to push a specific line, by choosing whether to publish r values or percent variance explained or graphs, they can pretty much hijack the intuitions even of people who don’t accept their reporting and try to rely on the numbers themselves.

The antidote is to have a good grasp of what each statistic means. And another antidote is to dial down your expectations. Remember, the study above was only able to correlate state IQ and state GDP at r = 0.4, but almost nothing in social science ever gets above 0.4. Trying to correlate rich parents with kids who become rich only got 0.2! 0.4 is pretty impressive and if you’re holding out for too much more you’re going to be living in a constant state of disappointment. I can think of one exception off the top of my head, and I am proud to say you will only find it here.