Education "reformers" from the Republican governors that support Michelle Rhee these days to Secretary of Education Arne Duncan back a whole host of measures allegedly aimed at finding and rewarding the most effective teachers, claiming that's the path to better-educated kids. I say "allegedly" for several reasons. In the case of Rhee and her nearest and dearest, it's really all about union-busting, as we've seen time and time again. Others may more sincerely believe that finding the best teachers and giving them bonuses while getting rid of the worst teachers is the path to a better educational system. It even has the benefit of seeming like completely obvious common sense. The problem is that the research really isn't there to back up this theory, yet federal and state funding is flowing toward it nonetheless.

The Education Writers Association has a helpful roundup of research on teacher effectiveness (PDF) that shows, most of all, how off-target the dominant discourse of "reform" is and how little we know about what makes a difference in schools.

First, one thing the evidence is completely clear on is that the most important factors for student achievement aren't found in schools. Income, parents' education, and other family and background factors account for an estimated "60 percent of variation in student achievement." That doesn't mean it's not worth figuring out how to make schools better, but it means that changing schools can only have limited effects, and even though teachers are generally agreed to be the most important within-school factor in achievement, estimates put their influence below 10 percent.

But the so-called reformers aren't targeting child poverty or other out-of-school factors. Still, we can agree that while changes to schools will only do so much, and reforms targeting teachers will only account for a part of what can be done in the schools, if teachers are the most important factor in the school, then they must have ongoing effects on student achievement, right?

About that. Basically, what studies tell us adds up to a big pile of "we don't know." Studies are conflicting and many just don't show much in the way of results for any of the things they're trying to test. There's a lot of measurement error.

Yes, students show greater gains under some teachers than others. But those gains tend not to persist when the students move on to the next grade. And while it's likely that in real life there are some teachers who, year after year, knock it out of the park, studies show a fair amount of variation in the test results that individual teachers get from one year to another; whether because of their own characteristics or measurement error we can't know for sure. Matthew Di Carlo at Shankerblog points to a study in which "only about 25-40 percent of the top quintile (top 20%) teachers in one year were in the top quintile the next year, while between 20-30 percent of them ended up in the bottom 40%," of which volatility "a very large proportion was due to nothing more than random error."

In some areas, like math, subject-area knowledge seems to make some difference. Certification has shown some benefits, again especially in math (where in-school factors seem to matter more than in reading). Merit pay rewarding teachers whose students do well, on the other hand, has not proven effective. Value-added measures are unproven.

In this litany of weak effects and no effects and uncertainty, one of the things we can say most confidently is that teacher experience matters. Teachers become steadily more effective for at least their first five years on the job and possibly beyond. Yet "reformers" like to target protections for teachers with seniority, arguing that it's the fresh-faced, inexperienced (and, no doubt coincidentally, lowest-paid) teachers who are best for students.

The point of running through all of this depressing and uncertain data is not to say that teachers don't matter. I'm not joining the Heritage Foundation and American Enterprise Institute in saying that teachers shouldn't be valued because they're stupid. The point is that the tests and measures we have right now don't tell us exactly what about teachers matters, or how to improve it. In fact, they don't even reliably tell us which teachers are doing the best by the standards of the tests and measures themselves. Add to that the fact that those standards aren't necessarily what we want to be building our education system around, anyway—math and reading are not all there is to a good education.

We should try to find out what makes a good teacher and how to make more teachers better. But that effort should include more of what goes on in the classroom every day and rely less on unproven tests. It should build on what we do know makes a difference and should embrace a diversity of subjects, not just two that appear on tests. It should acknowledge that education doesn't begin and end in the classroom, and should try to improve student achievement by improving students' lives.

And before anyone, "reformer" or otherwise, is allowed to call their approach data-driven, they should be made to confront the data (or lack thereof) on the effectiveness of the approach itself.