PART V: An FAQ for parents, students, and counselors

Compass has prepared this FAQ to further explain how schools and students can interpret the 2019 PSAT results.

Don’t scores fluctuate? Isn’t the standard error of measurement more than 20–30 points?

The overall drop on PSAT scores should not be confused with the standard error of measurement (SEM) that is inherent to any standardized test. SEM is an estimate of how much an individual student’s score may vary from form to form. Rather than view a 1340 as a completely accurate measure of ability, students are encouraged to think of a range of 1300–1380, reflecting the 40 point SEM for total scores on the PSAT.

SEM would be like a single golfer who averages 200-yard drives hitting the ball 190 yards some times and 210 yards other times. In testing jargon, there is a difference between his observed drives and his true driving ability. The 2019 PSAT was like a bad batch of golf balls leaving nearly every drive about 30 yards short. On the 2019 PSAT, lower scores impacted hundreds of thousands of students. The standard error of measurement still exists—it’s just now layered on top of a flawed scale. A shift of this magnitude would not happen with a well-designed exam.

Are percentiles still correct? Don’t they reflect actual performance?

No and no. The User Percentiles reported by College Board are not at all related to the scores received on the 2019 PSAT/NMSQT. Instead, the figures are aggregates from the 2016, 2017, and 2018 administrations. The User Percentiles for above average scores, therefore, under-report students’ actual relative standing on the 2019 exam. Once more, we can’t express this exactly without more data from College Board. Also, percentiles are never differentiated by form.

A rough rule-of-thumb for October 16th PSAT takers at 1200 and above would be to add 20–30 points to the total score and use the percentile tables from PSAT/NMSQT Understanding Scores 2019.



Should high schools treat scores differently?

First, schools should not be surprised to find that they have fewer high scorers this year. In our audit, we found a decrease of almost 40% in students scoring 680 or higher on the Math PSAT. The charts we provide in the Deep Data Dive section below give a rough sense of how things may have shifted. We will update our guidance if we are able to gather additional information. We now expect that most—if not all—states will see lower National Merit Semifinalist cutoffs. Based on our analysis, many states will see cutoffs 2–3 points lower than those seen for the class of 2020.

As a rule of thumb, schools offering the October 16th exam may want to add 20 points to Math scores above 600 and 10 points to ERW scores above 600 to bring them into closer agreement with previous years’ results.

Does this change how a student should choose between the SAT and ACT?

In most cases, the lower PSAT scores will not change a student’s decision. Any given test administration produces only an approximation of a student’s “true score.” Many factors come into play when deciding between the SAT and ACT. We would recommend that, when using a tool such as Compass’s PSAT/ACT Score Comparison or Concordance Tool, that October 16th students add 20–30 points to their scores if they are at a 1200 or higher. Unfortunately, we don’t have enough data to make a recommendation for students who took other administrations. A student who feels the PSAT results are unreliable should take an official practice SAT.

If the 2018 and 2019 PSATs are so different, could it be the 2018 test that was off?

The results from the 2018 PSAT were in line with those seen in 2016 and 2017. This seems to be a problem with the 2019 PSAT.

What lessons can students learn?

The job of College Board (and ACT) is to provide a fair testing environment by using parallel forms. One exam should not give students an advantage over students taking another. Unfortunately, this goal is not always achieved. While students have no way of predicting when a “bad” test will be offered, they can diminish its impact by spreading their risk. Relying on a single exam is poor planning. A student may perform poorly because of lack of sleep, a noisy room, an untrained proctor, an odd question or passage that trips up timing, a mismatch of skills studied and the problems appearing on that form, and many other reasons. The 2019 PSAT shows that students can’t even necessarily trust that the exams themselves are accurately scaled. An effective test plan must allow for the unexpected.

An alternative plan is to look more seriously at test optional or test flexible colleges. It’s unfortunate that students should plan for more testing to compensate for the vagaries of the exams.

Could the change be intentional? Were scores getting too high?

College Board has not announced any adjustments to the PSAT scale. Changing a scale without a specific rationale and without notice would be unprecedented. College Board would have little interest in lowering scores. We hope that College Board will clarify what went wrong with this year’s exam, but we don’t see any validity to this particular conspiracy theory.

Is any of this related to the introduction of experimental sections on the PSAT?

At schools not willing to pay additional fees, students were given “experimental” sections that are typically used to help scale and develop future tests. The use of these experimental sections may be evidence that College Board is taking the production and scaling of the PSAT more seriously. Any added diligence has not yet shown up in PSAT quality. The experimental sections are unlikely to have caused the scoring problem.

So what did cause the scoring mistake?

Only College Board can properly answer this question, and they declined to respond to this report. The most likely cause is poor sampling when developing the exam’s scale. If the sample group differed in unexpected ways from the reference group—differing in expected ways can be modeled—then the scale would have been incorrect. This is more likely to happen at the high end of the scale, where there are fewer test takers and fewer students in a sample group. Mistakes such as this one would spur most organizations to re-examine their procedures. Organizations can use a “red team” to intentionally probe for weaknesses or errors. College Board should have high standards of accuracy or at least be candid with students, schools, and colleges about what standards it does maintain.