Computer glitch could have misscored thousands of ISTEP tests, scoring supervisors say

Scores on thousands of student exams could be incorrect because of a computer malfunction that inadvertently changed grades on Indiana’s high-stakes ISTEP test, according to scoring supervisors familiar with the glitch.

But the company that scored the exam on behalf of the state — testing giant CTB McGraw Hill — decided to leave those potentially faulty scores in place, even after the problem was brought to management’s attention, the supervisors said.

Company executives would not speak with The Indianapolis Star, but in a letter Tuesday to the Indiana Department of Education, Executive Vice President Ellen Haley downplayed the problem. She said the issue “was very rare” and “did not affect student scores.”

Seven supervisors who spoke with The Star disagreed. All said they believed the problem was more widespread. Two estimated that tens of thousands of test questions were likely given incorrect scores. Others said it is difficult to put a number on the problem, but it was pervasive enough to merit rescoring the potentially impacted tests.

The integrity of ISTEP scores is critical because they are used to evaluate teacher and school performance. False results could affect teacher pay and lead to state takeovers of low-performing schools.

CTB is currently finishing the last step in grading last spring’s ISTEP ahead of the scores’ public release next month. The company is in the final month of a $23.9 million state contract to administer last school year’s test.

The potentially inaccurate grades are just the latest — though perhaps the most serious — issue to plague ISTEP, which is administered annually to more than 500,000 Indiana students in grades 3-8. Computer disruptions, grading delays and a political fight over the length of the test have made ISTEP a major source of contention in recent years.

State Superintendent of Public Instruction Glenda Ritz would not agree to an interview with The Star about the grading problems. But her spokesman, Daniel Altman, said he did not want to speculate on the accuracy of the test scores given what supervisors told The Star.

“We are going to obviously work with CTB to ensure that every single student gets the score that they earned on the test,” Altman said.

The Department of Education asked CTB to investigate the problem after receiving an anonymous letter on Nov. 25, Altman said. In response, CTB sent Haley’s Dec. 8 letter to the department assuring that student scores were unaffected.

Following questions from The Star, the Department of Education sent an email to CTB on Thursday, asking for the number of test items and schools that could have been affected by the computer malfunction.

The malfunction

The seven scoring supervisors who spoke with The Star — including some who worked on ISTEP — are temporary employees, but work full-time shifts and often oversee more than 100 evaluators who grade tests. They spoke on condition of anonymity, either because they signed confidentiality agreements regarding their work or feared losing their jobs.

They said they were shocked and dismayed by what happened at CTB’s Michigan Road scoring center earlier this year.

An evaluator discovered the computer problems on April 22, eight days after CTB began scoring ISTEP. The glitch affected open-ended questions that are each assigned two scores during the grading process, supervisors told The Star.

When the evaluator used the computer’s keypad to enter the two scores in quick succession, the second score would replace the first score. So a test question that should have received a score of 6 and 3 could end up with a score of 3 and 3.

After the glitch was discovered, managers told supervisors to have their evaluators use the mouse instead of the keypad to enter scores. But supervisors said they continued to see evaluators use the keypad.

More than a week after the problem was discovered, Mike Conarroe, the company’s director of hand scoring, held a meeting with supervisors.

At that April 30 meeting, Conarroe informed them that a fix was in progress and that evaluators should continue using the mouse instead of the keypad.

But when questions were raised about the integrity of the exams already scored, Conarroe became agitated and said the company was not going to redo eight days of work, according to the seven supervisors who spoke with The Star and were present at the meeting.

Conarroe promised future meetings about the issue — but that never happened, the supervisors said.

Not until early May was the keypad disabled in an effort to prevent incorrect scores, according to Haley’s letter to the department.

The decision not to rescore the tests prior to that deeply troubled the supervisors who spoke to The Star. One said 1.2 million individual responses were graded during the eight days before the malfunction was discovered. Redoing that work, some supervisors estimated, would have cost the company at least $500,000. The decision not to rescore came less than two months after the state dropped CTB as its ISTEP vendor for 2016, some supervisors noted.

In explaining why they agreed to speak with The Star, they said the company’s actions violated a basic tenet of standardized testing — that every student’s score should accurately reflect academic achievement. They also expressed concerns about the effect on teacher pay and schools.

Some said the decision weighed on them personally. One said he had trouble sleeping at night.

CTB’s response

Conarroe declined to speak with The Star, referring reporters to the corporate communications office. Corporate spokespeople for CTB McGraw Hill and the company that purchased its assessment scoring business earlier this year, Data Recognition Corp., did not respond to phone calls from The Star.

But in the company’s letter to the Department of Education, Haley said the company’s data monitoring and double-checks of some individually scored questions did not indicate an impact on student scores.

“The issue was not a common occurrence, was actually difficult to create in live scoring, and was fixed quickly — both in on-the-floor instructions and then technically in the PEMS software,” she wrote. “Based on CTB’s quality control tests, there was no need to rescore any tests as a result of the keypad issue.”

She also said most evaluators use the mouse instead of the keypad and don’t enter scores in rapid succession because they are trained to reread answers to questions before entering the second score. As a result, “very rapid entry was highly unusual,” she wrote.

But supervisors said the keypad is the tool of choice among evaluators because it allows rapid entry of scores, which is a top priority for the company. They also said most evaluators only read a student response once, then assign both scores, one immediately after the other.

A history of problems

Local school officials are already upset about a steep drop in scores amid a move to new, more rigorous standards. In recent weeks they have described the ISTEP process as a “complete fiasco” and a “botched system.”

And that was before the legitimacy of the scores was in question.

What role, if any, the computer malfunction may have played in the lower outcomes is unclear. But some school administrators say what is evident is that the system is flawed.

“We are just assuming that CTB is giving us the correct information,” Warren Township Schools Superintendent Dena Cushenberry said. “There is no checks and balances to the system.”

It’s not the first time scoring problems have caused concerns.

In August, CTB said it was facing a delay in the scoring of new technology-enhanced items. Then in November, the state decided to award some students bonus points because of differences in difficulty in the paper/pencil and online versions of the test.

And that was just this year.

The state recovered $3.3 million in credits and in-kind services from CTB after testing disruptions during the 2013 ISTEP. Altman said the Department of Education is also pursuing damages for other problems.

The state announced in March that it would drop CTB for the 2016 ISTEP and use Pearson Education instead, even though that company has had problems of its own in other states.

CTB is now in the process of rescoring tests — a normal part of the process where parents whose students failed can request a second look. Such requests doubled this year. The window for parents to make such requests closed Nov. 13.

Even if just one student’s score was wrong, that is too many, they said. It’s a position the company should agree with, given the poster that hangs on the scoring center’s wall.

“Accuracy is always our number one priority,” it reads. “Remember that each book you score represents an individual student.”

Call Star reporter Chelsea Schneider at (317) 444-6077. Follow her on Twitter: @IndyStarChelsea. Call Star reporter Tony Cook at (317) 444-6081. Follow him on Twitter: @indystartony.