HA note: The following is reprinted with permission from Kathryn Brightbill’s blog The Life and Opinions of Kathryn Elizabeth, Person. It was originally published on July 24, 2013.

When I was in college, I was one of the participants who answered the NHERI/HSLDA/Brian Ray survey of homeschooling graduates. I don’t remember how I got the survey, probably via email forward from my mom, but what I do remember is sending an email to my family after taking it in which I said something about how I wouldn’t believe a thing from the eventual study results. Aside from the fact that an email forwarded among homeschool groups asking people to take an internet survey is a lousy way of getting a representative sample (especially when we’re talking more than a decade ago, when we were nowhere as close to ubiquitous computing as we are now and computer access was still largely along socioeconomic lines), the survey itself was rife with methodological problems.

My memory is of a survey where I could tell exactly what answer they were looking to find, based on both the questions asked and the possible answers given for those questions. It was a survey that, even as a college student who had a positive experience and didn’t have the criticisms of the system that I do now, was so bad that I wondered why anyone would design it the way they did unless the goal was not usable data but a certain set of predetermined results.

If I got anything out of being homeschooled for twelve years by a math teacher, it’s a healthy appreciation for numbers. Numbers explain the world, or at least they can explain the world if they’re used correctly. Without good numbers, you might as well be stumbling around in the dark bumping your shins into the furniture. Bad numbers are even worse than no numbers; they’re like spreading Legos on the carpet while stumbling around in the dark with bare feet.

The HSLDA/NHERI numbers on homeschooling are like stumbling around on that Lego-strewn carpet. Aside from the aforementioned homeschool graduate survey, they also have test scores that they trot out to prove that homeschoolers do better on standardized tests than other students.

Except that the numbers don’t show that. What they actually show is the results of those homeschooled children whose parents not only use achievement tests (which isn’t all of them), who were aware of the NHERI survey (again, not all of them), and who agreed to submit their children’s scores for a study that they were already told was intended to make homeschooling look good. We have no idea how representative that group is of homeschoolers as a whole because it was self-selected and we don’t know whether that self-selected group in any way mirrors homeschoolers as a whole.

The modern homeschool movement is more than 30 years old and we still don’t have a good idea of homeschooling demographics. We know that they’re predominantly white and might be mostly religious (but even the religious makeup is hard to guess because the religious and the secular homeschoolers seldom run in the same circles). We don’t know what the socioeconomic makeup of the homeschool population is, and we don’t know the average education level of parents or a breakdown of those parents’ degrees. Without that, we can only guess at whether the socioeconomic levels of homeschoolers in NHERI’s test score data in any way represent homeschoolers as a whole.

Homeschoolers are fiercely independent even without taking into consideration that the religious ones don’t trust outside researchers because they’ve been told by HSLDA for years that outsiders want to hurt homeschooling. As for the secular unschoolers, they aren’t exactly fans of the establishment, which makes it hard to pin them down as well. Compounding the problem, the people the religious ones do trust (aka, HSLDA and NHERI) aren’t interested in giving good data, they’re interested in giving a good sales pitch for homeschooling. And so, we’re stumbling around in the dark knocking our shins into furniture and stepping on stray Legos.

There are a handful of small studies about adult former homeschoolers, but the rest is guesswork. We don’t have a good picture of the socioeconomic status of homeschooling families. We don’t really know how homeschoolers as a population do on achievement tests because we don’t have any surveys that aren’t self-selected. We don’t even have a good idea of how many people are homeschooling. It might be a million and a half, it might be twice that. We need good data and we simply don’t have it.

Are the kids I knew in high school who were all upper middle class children, mostly of parents with degrees in STEM fields, an accurate representation? What about the kids I knew when I was younger whose blue collar parents had no college and were passing their non-standard English dialect on to the kids rather than standard English grammar? Was my dad right when he always said that math was a weakness that homeschoolers needed to watch out for because the kids were being taught primarily by mothers who had been put on the consumer math track in school? Was I right when I used to tell people that the reason they thought homeschoolers were weird was because the normal ones who are in the majority didn’t advertise that they were homeschooled?

I don’t know because no one does.

And yet the Brian Ray/NHERI research keeps getting repeated in conservative and mainstream media even though it doesn’t actually tell us anything.

I understand why homeschool parents want to like the NHERI numbers; they help reassure them that they’re on the right path and not totally screwing up their kids with the homeschool choice. That’s why what NHERI’s doing with self-selected, unscientific research methods is so messed up. Parents, good parents, are relying on data that’s given a veneer of science but that may well be leading them to make educational choices that aren’t the best possible for their children. If math education, for example, really is the weakness for homeschoolers that my dad thought it was, homeschoolers need to know that so they can compensate for the weakness. If more than a fringe number of homeschool kids are having trouble fitting in with mainstream society because of the homeschool bubble, parents need to know that to correct for it.

With nothing but bad data to rely on, parents are left stumbling around in the dark hoping that they aren’t going to bash their shin on the corner of the coffee table or step on a stray Lego. That’s not serving homeschool parents well, and it’s certainly not doing anything for the kids.