Follow me on Twitter at @AnthonyCody

The New York Daily News has perhaps inadvertently shed some light on why teachers might be hesitant to have a large portion of their evaluations based on standardized test scores. In a rare moment of transparency, one of the 8th grade reading comprehension questions has been published, in a story broken by Leonie Haimson on the New York Parents blog, and it has many people scratching their heads.

The story is an absurd tale of a talking pineapple, who challenges a hare to a race. The story must be read to comprehend the controversy.

The questions that follow make even less sense. Students are asked

1. Why did the animals eat the pineapple? a. they were annoyed b. they were amused c. they were hungry d. they wanted to

2. Who was the wisest? a. the hare b. moose c. crow d. owl

Pearson, the largest test publisher in the world, has said that company policy forbids them from commenting.

This story is remarkable for several reasons. First of all, the patent absurdity of the questions. In today's data-driven world, the scores these student achieve could be used to end a teacher's career.

But there are many test questions that are of questionable quality. Scientist Robert Krampf this week also found major errors in Florida's Science FCAT test.

He writes:

The Science FCAT is Florida's high stakes test that assesses all the science concepts and information that students should have learned by the end of fifth grade. Schools and districts are subject to financial incentives or penalties, depending on their students' FCAT scores, so this is a VERY important test.

The test was developed using guidelines that provide the basic information the test is supposed to be based upon. The glossary includes the following definition:

Predator--An organism that obtains nutrients from other organisms.

Dr. Krampf points out,



By that definition, cows are predators because they obtain nutrients from plants. The plants are predators too, since they obtain nutrients from decaying remains of other organisms. I have yet to find anyone who thinks that this is a proper definition of a predator.

He goes on to enumerate many other errors.

In these circumstances, the safest course of action for a science teacher in Florida is to make sure his students memorize bogus information, because that is what they will be tested upon.

I do not know what the teachers in New York can do to prepare their students for the pineapple story - perhaps have them watch some episodes of Monty Python.

Teachers who give standardized tests are required to sign affidavits swearing they will not copy the tests, or divulge their contents. Thus teachers are forbidden from airing concerns they might have about the contents of the tests.

The tests have become the ultimate authorities in our schools, and the test publishers are virtually unquestionable.

The standardized testing technocracy has convinced our policy makers that the only way we will be competitive in the world is if everyone learns the same information, and has that learning measured in ever-finer increments. We are not supposed to look behind the curtain to see the way this data is arrived at.

We are promised that any problems in the system will be fixed by the next generation, the Common Core, the computers that can score tests as well as the current system of warehouses of poorly paid readers now used for that purpose.

The truth is that sensitive formative assessment is the proper domain of a well-trained, intelligent teacher, capable of seeing the individual strengths and weaknesses of children, and guiding their learning. Standardized tests are useful when used as an annual check on that learning, but that is all. Once heavy consequences are attached to them, all the learning in a classroom is re-oriented to focus on pleasing that master, that almighty unquestionable arbiter of what has been learned.



Teachers can be questioned, and even challenged. I love it when a student disagrees with something I have told them. This is the best sort of teachable moment, because the whole class becomes engaged in trying to find out the truth. The best way I have found to handle this is to make the question a homework assignment, and invite everyone to investigate. And if I am wrong, that is fine by me. I do not need to be the ultimate authority. I want my students to challenge me, and anyone they encounter, if they think they are wrong.

But teachers cannot discuss publicly, let alone challenge the content of the tests, and the test publishers will not even discuss it when the questions are somehow made public. How long will we pretend that this is any way to teach our students to be critical, creative thinkers?

Update: New York Education Commissioner John King has released a statement that says:





First of all, the "passage" printed in the media is not complete. Although the questions make more sense in the context of the full passage, due to the ambiguous nature of the test questions the Department has decided it will not be counted against students in their scores.



It is important to note that this test section does not incorporate the Common Core and other improvements to test quality currently underway. This year's tests incorporate a small number of Common Core field test questions. Next year's test will be fully aligned with the Common Core.



This particular passage, like all test questions, was reviewed by a committee comprised of teachers from across the state, but it was not crafted for New York State. It's a passage that has been used in other states and was included by Pearson Inc., the test vendor, to provide a comparison between New York students and students from other states.1 The passage and related questions are not reflective of the precision of the entire exam.



The accuracy and efficacy of our state assessments are crucial to our reform efforts and measuring student academic growth. We will, as always, review and analyze all questions on every assessment we administer.

You can see the actual text as used in the test here.

Seems like he is saying, "Move along, nothing to see folks. The Common Core will make it all better. Trust your authorities."



Update 2: Daniel Pinkwater, the author of the Pineapple story has commented on its use here, saying,

It's hilarious on the face of it that anybody creating a test would use a passage of mine, because I'm an advocate of nonsense. I believe that things mean things, but they don't have assigned meanings.

He went on to say:

I might have said, "They're making a dishonest living doing these tests, but they're doing no harm." But maybe now they are. And certainly they're sucking up a lot of money that could be put to better uses in education, and I think the whole thing is shameful.

What do you think? Are test publishers worthy of the authority we are now vesting in them?