Offering big bonuses to teachers failed to raise students’ test scores in a three-year study released Sept. 21 that calls into question the Obama administration’s push for merit pay to improve education.

The study, conducted in the metropolitan Nashville school system by Vanderbilt University’s National Center on Performance Incentives, was described by the researchers as the nation’s first scientifically rigorous look at the effects of merit pay for teachers.

It found that students whose teachers were offered bonuses of up to $15,000 a year for improved test scores registered the same gains on standardized exams as those whose teachers were given no such incentives.

“I think most people agree today that the current way in which we compensate teachers is broken,” said Matthew Springer, executive director of the Vanderbilt center and lead researcher on the study. “But we don’t know what the better way is yet.”

The study comes as the Obama administration is encouraging school systems to link teacher pay and tenure to how students perform on tests and other measures of achievement.

The researchers looked at fifth- through eighth-grade math teachers from 2007 to 2009. A group of about 300 teachers started out in the study; half were eligible for the bonuses, the other half were not.

The bonuses were given out based on improvements in scores on Tennessee’s standardized exam, which is used by the state as part of the federal No Child Left Behind requirements.

Springer was quick to point out that his study looked only at individual bonuses, not extra pay doled out to teams of teachers or an entire school. He said more research is needed before policy makers draw any definitive conclusions.

“Some people were initially disappointed when they saw the results, but quickly turned around and said, ‘Well, at least we finally have an answer,'” he said. “It means pay can’t do it alone.”

The federal Education Department called the study too narrowly focused.

“It only looked at the narrow question of whether more pay motivates teachers to try harder,” said spokeswoman Sandra Abrevaya. “What we are trying to do is change the culture of teaching by giving all educators the feedback they need to get better while rewarding and incentivizing the best to teach in high-need schools [and] hard-to-staff subjects.”

The American Federation of Teachers praised the study and argued that teachers need other resources, including better training and more supportive administrators.

“Merit pay is not the panacea that some would like it to be. There are no quick fixes in education,” said union president Randi Weingarten. “Providing individual bonuses for teachers standing alone does not work.”

Teachers unions have historically opposed merit pay, arguing that test scores are not an accurate measure of student achievement, that financial rewards could pit teachers against each other, and that administrators could use bonuses to reward favorites and punish others.

Jennifer Conboy, a high school social studies teacher in Miami, called merit pay a “baseless fad.”

“Merit pay is an excuse to resist the attempt of teachers to get fair pay in the first place,” the 37-year-old Conboy said. “On a personal level, merit pay would do nothing to me. I took this job because I think education is the bedrock of a functioning democracy, and if I cared about democracy—which I do—then I had a responsibility to do whatever I could to strengthen education.”

Only a few schools and districts across the country have merit pay, and in some states the idea is effectively illegal. The Obama administration hoped to encourage more states to pass merit pay laws with its $4.35 billion “Race to the Top” grant competition.