This study aims to understand the effectiveness of typical admissions criteria in identifying students who will complete the Physics Ph.D. Multivariate statistical analysis of roughly one in eight physics Ph.D. students from 2000 to 2010 indicates that the traditional admissions metrics of undergraduate grade point average (GPA) and the Graduate Record Examination (GRE) Quantitative, Verbal, and Physics Subject Tests do not predict completion as effectively admissions committees presume. Significant associations with completion were found for undergraduate GPA in all models and for GRE Quantitative in two of four studied models; GRE Physics and GRE Verbal were not significant in any model. It is notable that completion changed by less than 10% for U.S. physics major test takers scoring in the 10th versus 90th percentile on the Quantitative test. Aside from these limitations in predicting Ph.D. completion overall, overreliance on GRE scores in admissions processes also selects against underrepresented groups.

INTRODUCTION

Physics is the least diverse of the sciences, rivaling mechanical engineering and aerospace engineering for the least diverse fields within all of science, technology, mathematics, and engineering (STEM) (1). Groups underrepresented in physics include Blacks, Latinos, Native Americans, and women of all racial/ethnic groups. Barely 5% of physics Ph.D.’s are granted annually to those identifying with an underrepresented racial/ethnic category; women earn only 20% of physics Ph.D.’s. The origins of these vast representation gaps are complex and include inequitable educational access from an early age (2), implicit bias in the classroom and research laboratories (3), deterrents to continuation for underrepresented groups (e.g., departmental climate and disciplinary culture) (4–6), and stereotype threat (7, 8). Expanding gender and racial participation in STEM is important for the development of a robust domestic scientific workforce, however, as pointed out by the National Academy of Sciences report Expanding Underrepresented Minority Participation: America’s Science and Technology Talent at the Crossroads (9). Who gets to do the science of the future is determined largely by who is selected into Ph.D. programs. Transition of students to graduate work is thus a concern of national importance; only by attending to structural issues present in the process of selecting who gets to do the science of the future can we make sustainable progress toward broadening the participation of groups historically underrepresented in STEM.

Unfortunately, nontrivial barriers impede admission to Ph.D. programs for some demographic groups. Undergraduate grades, college selectivity, and GRE scores are the three criteria that best predict admission to U.S. graduate programs (10), but these parameters are not evenly distributed by race and gender (10, 11). This situation is particularly problematic for easily sortable numeric metrics, such as GRE scores. Predictive validity analyses of the GRE are almost as old as the test itself (12–14). Research over decades of test refinement, as well as meta-analysis of this research, consistently finds that scores on the Verbal and Quantitative GRE (GRE-V and GRE-Q, respectively) have weaker validity for Ph.D. attainment than for graduate school grades (15). Using the same database as (15), additional analysis identified positive relationships between these tests’ scores and first-year grades, cumulative grades, and faculty ratings (16). In a similar vein, two recent studies on biomedical Ph.D. admissions found that the General GRE does not predict scholarly productivity (17) or degree completion but that scores are associated with first-semester and cumulative graduate school grades (18). Methodologically, most assessments of validity focus on the general test and are limited to bivariate correlation analyses; they do not include covariates to render more precise estimates. Overall, the record indicates that the GRE’s validity wanes as time elapses between taking the test and measuring “success” in graduate school, which may be indicated by completion, research productivity, and other markers of success.

Despite their near-universal employment by physics Ph.D. programs (19), no study has tested the validity of common admissions metrics explicitly in these programs. Given the strong race, gender, and citizenship performance differences on the GREs in particular (10, 11), it is critical that we know the extent to which scores are useful in identifying students who will complete the Ph.D. We conducted such a study, inviting physics programs to submit de-identified student admission and degree completion records. Among applicants who were admitted and matriculated into physics Ph.D. programs, we find the predictive validity to be poor for some of the most ubiquitously used admissions criteria. In particular, we find undergraduate GPA (UGPA) to be the most robust numerical predictor of Ph.D. completion, and, despite a large sample size and wide dynamic range, we do not find a statistically significant relationship between GRE Physics (GRE-P) Subject Test scores and Ph.D. completion.

This article is structured as follows. First, we provide a snapshot of the state of U.S. physics with respect to diversity and degree production. Next, we describe U.S. citizens’ performance on the GRE-P across a variety of demographic parameters. We then describe our multivariate regression analysis and its findings. Last, we conclude with implications of these results.

Current state of U.S. physics The state of diversity in physics can be summarized by the annual average numbers of bachelor’s degrees awarded, first year graduate students, Ph.D.’s awarded, and the performance of students on the GRE-P. Whereas the latter data are obtained from ETS itself, the remainder are available through the Integrated Postsecondary Education Data System (IPEDS) (20). From the IPEDS data (Table 1), several observations are possible. At all stages of physics education, Latinos and Blacks are underrepresented relative to their college-age representation in the United States, whereas Asians and Whites are overrepresented. The ratio of GRE-P test takers to physics undergraduate degrees awarded indicates that approximately half of physics undergraduate degree earners are actively considering physics graduate studies. About one quarter of U.S. physics majors matriculate into U.S. physics graduate programs. Significant exceptions to these trends are noted for Blacks, who take the GRE-P and matriculate at lower rates than the national average. Black females, Latinas, and Native Americans of any gender each had fewer than 10 physics Ph.D. matriculants annually in these data. Women are barely 20% of physics students, at both the undergraduate and graduate levels, and they take the GRE-P in proportion to their representation. Table 1 Multiyear averages for diversity metrics in U.S. physics. Each row’s entries show the percentage of the U.S. annual average in the second column. Data for the GRE-P are the average over test years 2009–2015; all others are the averages from 2009–2014 (20). We excluded data if an entry’s absolute number was 10 or less (indicated by asterisks) and excluded race categories “other,” “two or more,” and “no response.” F, female; M, male. View this table: IPEDS data indicate that around 60% of U.S. citizens who matriculate to Ph.D. programs will complete their degree. We do not take into consideration the time dependence of matriculants and Ph.D.’s earned, leading to some ambiguity in completion rate. However, we note that the Council of Graduate Schools indicates that the 10-year completion rate for physics overall is 59%, close to what we report here (21). There is no overall gender gap for physics Ph.D. completion or time to Ph.D. among U.S. citizen graduate students. With the caveat that low enrollment numbers imply a relative error on the order of 20%, these data indicate that Hispanic males have a lower Ph.D. completion rate than the average and that Asian females have the highest Ph.D. completion rate of U.S. citizens (Table 1). Figure 1 shows significant gaps in GRE-P scores for U.S. citizens based on race and gender. These data, obtained from the ETS database (portal.ets.org), represent all test takers who earned a valid GRE-P score in test years 2009–2015. The median U.S. female score is 580 (28th percentile), while the median U.S. male score is 650 (46th percentile); ETS reports (22) the SE of measurement to be 49 points (roughly 9th percentile), indicating that gender gaps are statistically significant. Notably, similarly large gender gaps in GRE-P scores exist for all racial/ethnic groups for both U.S. and international test takers [median percentile by country for (male, female) test takers were as follows: China (86th, 77th), India (70th, 46th), and Iran (62nd, 42nd)]. The median scores for Black (530; 17th percentile), Hispanic (580; 28th percentile), White (630; 39th percentile), and Asian Americans (690; 53rd percentile) also reveal significant variation in GRE-P by race. Fig. 1 The fraction of U.S. test takers above a specified GRE-P score shows that cutoff scores adversely affect underrepresented groups more than majority groups. Given that we find that Ph.D. completion is not correlated with the GRE-P score, the misuse of the test in admissions will negatively affect diversity without being able to identify individuals able to complete a physics Ph.D. Source: ETS. Although the best evidence suggests that faculty are well intentioned when selecting students, many are unaware of demographic patterns in GRE scores and they carry out admissions according to inherited practices that include using cutoff scores (23). Programs using the GRE-P as an integral part of their admissions process may be unwittingly selecting against underrepresented groups and U.S. citizens. This effect is easily inferred when combining the race, gender, and citizenship score differences with the use of strict cutoffs (or even preferences) based on GRE scores. Unfortunately, use of minimum acceptable GRE scores in graduate admissions is a common practice throughout the disciplines (23) and in physics specifically (19, 24). Approximately 25% of physics Ph.D. programs publicize to potential applicants a minimum acceptable GRE-P score around 700 (55th percentile). The representation of the U.S. test takers above this level is very different from the applicant pool: Hispanics and Blacks are 6.2 and 1.8% of test takers, respectively, but only 4.1 and 0.6% of those whose scores exceed 700; Asians are 7.8% of test takers, but their above-700 representation is 11.4%; the representation of Whites is unchanged at 78%; women are 20% of test takers, but only 11% of those scoring above 700.