On April 4, 2017, emergency medicine (EM) applicants learned that required participation in the Standardized Video Interview (SVI) Project Study would be part of their residency applications. Applicants who do not complete the SVI will still be able to apply to emergency medicine, however, programs will be informed of their decision to not participate and applicants have been warned their application may not be considered “complete.” This suggests that while the process is not technically mandatory, it is in the sense that applicants may be penalized for choosing not to participate. The Emergency Medicine Standardized Video Interview Working Group (EMSVI), composed of AAMC and CORD, SAEM, EMRA, and AAEM representatives, moved forward with a second‐phase pilot project after roughly 600 students volunteered to participate in the original pilot during the previous application cycle, with participating students receiving financial compensation. The SVI interface, created by HireVue (a third‐party for‐profit entity), presents applicants with six sequential questions, allowing 30 seconds of preparation and 3 minutes to answer each prompt. After each 3‐minute recording, the next question begins with no opportunity for reviewing or repeating. The initial pilot study was an institutional review board (IRB)‐approved research study, while this current second phase is an “operational pilot,” without IRB input/review.1 These questions are designed to evaluate two ACGME competencies: “Professionalism” and “Interpersonal and Communication Skills.” Video responses scored by “trained third‐party raters” will be provided to residency programs as part of students’ applications and computer analysis of select populations will be conducted in parallel.2 We urge the AAMC and EMSVI working group to slow SVI implementation, share preliminary findings, and allow students to decline participation until formal student representation to the EMSVI working group has been created. Further, the AAMC should consider removing the SVI score from the ERAS application until its validity is longitudinally evaluated.

Validity We commend the rigorous efforts the AAMC has undertaken to acquire validity evidence regarding the SVI as an assessment tool. Pilot study data have been shared in a number of venues and addressed four primary research questions:3 Do raters demonstrate adequate agreement/reliability? Did raters use full range of the rating scale? Do ratings differ by sex and race/ethnicity? What is the correlation between SVI ratings and step examinations? These research questions, however, do not reflect whether or not SVI score are associated with true resident performance or interpersonal skills. A high or low score on the SVI can carry no meaning (nor should it) until an established correlation between SVI score and the ACGME competencies it hopes to evaluate is found. Regarding standards for developing validity,4 an argument that a low score on the SVI can “predict” which residents will struggle with professional behavior or that a high score can predict excellent interpersonal skills is years away. By reporting SVI scores within applications this year, programs are using an unvalidated metric to make meaningful professional decisions that impact applicants who would have otherwise received (or not received) an in‐person interview. Although the AAMC states, “An IRB‐approved research study can't provide … correlations between SVI scores and trainees’ performance,”3 we feel that robust validity evidence could be obtained within the framework of an IRB‐approved longitudinal study with previously enrolled applicants.

Necessity The SVI's goal is to evaluate the two ACGME competencies described above, which overlap directly with those competencies assessed by the Step 2 Clinical Skills (CS) examination. With similar objectives, to what extent is the SVI a redundant assessment of students’ abilities to behave professionally and demonstrate adequate interpersonal skills? If the SVI truly evaluates different characteristics than those already demonstrated by the Step 2 CS examination, we urge the AAMC to explain those differences and how the SVI contributes to the assessment of an applicant in the context of the objectives not satisfied by the Step 2 CS examination. Further, the Step 2 CS has “pass/fail” reporting to residency programs (rather than stratified scoring shared with applicants) indicating that they have been deemed proficient in these areas by a regulatory body. We wonder what the rationale is for trying to create a “score,” based on variables already assessed along this “pass/fail” continuum. Finally, with most EM applicants completing two to three EM rotations, we estimate that each has approximately 192 to 288 hours of observed EM performance in which their professional and interpersonal capabilities are discussed and stratified within SLOEs (twelve 8‐hour shifts per EM block/Sub‐I for two to three EM rotations). How much more is gained from 18 additional minutes?

Costs Students currently incur ERAS fees and away rotation expenses and pay for the majority of interview‐travel and accommodations. The AAMC has waived SVI costs to the applicant during this second‐phase pilot, but has not determined what expense to students the SVI will be in future years. The additional costs produced by utilizing third‐party, for‐profit companies should be discussed prior to continuing a mandatory pilot‐phase study. Has there been consideration to the ethics involved in creating an industry around the SVI? While the AAMC has stressed that the SVI requires no special preparation or audiovisual equipment, we are reluctant to believe this will hold true regarding our current culture of review guides/”first aid” for every step of the process from the clerkship to the interview. The EM application process should work to minimize the commodification of medical education, rather than add processes that require additional financial commitment. Consider also the time and “personal cost” of the application process on the applicant. Adding another obstacle to the application process is a point worth exploring; applicants already provide the following to be downloaded and stratified: 1) medical honor society status, 2) GPA/grades, 3) Step 1 score, 4) Step 2 CK score, 5) Step 2 CS pass/fail, 6) three to four SLOE/LORs, 7) research experience, 8) volunteer experience, 9) work experience, 10) the medical student performance evaluation, 11) personal statement, and 12) academic awards. Does the addition of a 13th data point provide enough value to justify these costs or enhance an applicant's portfolio beyond what has already been provided? Finally, we must also consider the cost and time that residency programs will themselves experience due to the implementation of the SVI. Program directors already must review hundreds (if not thousands) of applications per interview season. A significant time investment is necessary to review an application in entirety, especially when considering the importance of offering interviews to candidates that will be the best fit for each program. Programs will now be forced to either review dozens of hours of SVI footage or to rely on the reported numerical score with no precedent on how to use it. In addition to the time cost that reviewing SVI footage entails, current advertising foreshadows the consequences of “for‐profit” enterprises entering the medical education/residency application process to garner subscription purchases. Shortly after the announcement of the SVI, RIVS Video Interviews, another Web‐based technology company for digital voice and video interviews, approached EM program directors offering a discounted license on their RP86 video assessment tool to help programs “gain insights at the earlier stages of the match process into the soft skills of already technically qualified candidates.”

Credibility and Consequences Forget resumes and profile data, Insights analyzes over 15,000 interactions, hiring, and performance attributes. A data‐driven recommendation engine that predicts which candidates are most likely to be top performers. Predict your next performers and find them fast using your company's data!5 From what information has been provided by the AAMC/HireVue, sometimes a person, but potentially a computer assessment (via HireVue) will be used to review and assess the video interviews as the EMSVI study progresses. Utilizing psychometric analysis software to stratify human reaction and predict potential gives us pause. By reducing each applicant's “interview” to some ordinal measurement, the body language, tone, and personality that defines person‐to‐person interviews may simply devolve into yet another “test.” Potentially we are Luddites, but HireVue advertises the following: While we do not currently know if the “Insights,” software will be utilized in the final SVI tool, the AAMC has stated that they will be conducting a parallel research project to explore the “possibilities of computer scoring.” They have also acknowledged that if SVI were to expand to larger applicant pools, they would likely need to utilize this technology.1 Consider also the consequences for a student who simply makes a mistake: The SVI relies on unedited answers to create a score provided to all residency programs, creating a single high‐stakes scenario in which there is no opportunity for feedback or self‐reflection. In almost no other scenario does a single interview encounter potentially define an applicant's entire professional portfolio. In contrast, in‐person interviews are unique, providing bidirectional exchange of opinions, experiences, and linguistic styles that are adaptable. A poor interview can be used to learn and adapt for the next interview. The ability to adapt is itself a desirable quality, seemingly bypassed by the SVI, where mistakes made during recording impact the entire application.

Ethical Concerns and Bias This second‐phase pilot project is not an IRB‐approved study; however, it features many characteristics and outcome variables that usually fall within the purview of an IRB. We worry that this itself presents a serious ethical dilemma. Students are considered a special class of vulnerable populations due to three broad areas of concern: the risk for coercion given the undue pressure they will have to participate in research, the compromised relationship between students and educators due to the research, and the fact that research on medical students may pose risks that are not readily apparent to either the students or investigators.6, 7 The EMSVI second‐phase design presents several concerns related to participant coercion: The desire to have 100% of applicants complete the EMSVI and the score advertisement to all programs seemingly puts participants in a position where lack of an SVI score may be perceived as a “red flag” on their application. It is odd that the study, run by the AAMC (the governing body that oversees the entire application process) does little to quell this concern. And what of the consent process? The argument can be made that students are implicitly consenting to be studied by signing up for the SVI. However, can consent occur without coercion for a mandatory activity overseen by the very organization that controls the applicant's ability to get into residency? We have concerns that this second‐phase pilot would not be approved by the IRB in the form it exists now: a high‐consequence test with no validity evidence used in a vulnerable population that has little regulatory say, but a massive personal stake in the application process. This mandatory participation within a “pilot project” reduces students’ autonomy and protection from unknown potential implicit bias. Finally, it appears that there was no involvement by medical students in the formation of the EMSVI working group, and we feel that the student perspective is an invaluable resource. A decision impacting all medical students, made in isolation from those very medical students, represents poor precedent for creating solutions to a burdened application system. The EM residency match draws applicants from diverse domestic and global backgrounds introducing communication and cultural differences. While the AAMC is addressing potential bias by training both SVI evaluators and residency programs, little research exists on whether this is adequate to protect minority and underrepresented student populations. The addition of video to EM applications inherently introduces the potential for bias against qualities that themselves do not impact professionalism or communication. For example, the HireVue description of the SVI emphasizes the importance of body language; however, body language varies culturally and adds a significant confounder to the scoring system. In fact, the initial information presented from the first‐year pilot showed students that identify as Asian performed, on average, one point worse than their peers who identified as black or white, while those who identified as Hispanic were one‐half point behind self‐identified black and white applicants. These groups are critically important to a diverse and representative EM applicant pool, and attention to how the SVI impacts these applicants is an important step to any mandatory application component. Given that the SVI metrics must inherently weigh the qualities of certain applicants higher than others, the SVI platform discounts the many different ways an applicant's cultural and psychosocial qualities can lead to being a successful EM resident. This potentially homogenizes applicants and unfairly selects for certain characteristics over others.

Summary We believe that the strongest residencies feature trainees of diverse personalities and backgrounds. We believe that true understanding and excellence in patient care are nurtured by these differences, and we firmly disagree with any actions that, intentionally or not, diminish our field's diversity. Attempting to predict future potential from a psychometric present may be shade to growth for the diverse physicians that EM hopes to attract. If applicant metrics are not adequate or valid, we must reevaluate their utility and inclusion. However, we recommend that a more thorough explanation of the need for the SVI pilot study be shared, with consideration to the possibility that other markers of an applicant's portfolio may already satisfy this need. We question the utility of an additional measure that increases burden and potential bias on medical students who have no ability to decline participation or participate in design. Until the validity of the SVI's ability to predict future performance is determined, mandatory participation and reporting to residency programs should not be implemented.