Materials and Methods Survey methods and administration The data for this study came from an anonymous survey of international members of the STEM community using Qualtrics survey software. Data were collected under institutional review board agreements and federalwide assurance at Occidental College (IRB00009103, FWA00005302) and California State University, Northridge (IRB00001788, FWA00001335). The survey was administered between 28 February 2019 and 10 May 2019 using the online-based platform in English and was open to anyone self-identifying as a member of the STEM community that published their scholarly work in a peer reviewed system. Participation in the survey was voluntary and no compensation or incentives were offered. All respondents had to certify that they were 18 years or older and read the informed consent statement; participants were able to exit the survey at any point without consequence. Participants were recruited broadly through social media platforms, direct posting on scientific list-serves and email invitations to colleagues, department chairs and organizations focused on diversity and inclusive practices in STEM fields (see Supplemental Files for distribution methods). Targeted emails were used to increase representation of respondents. Data on response rates from specific platforms were not collected. The survey required participants to provide basic demographic information including gender identity, level of education, career stage, country of residence, field of expertise and racial and/or ethnic identities (see Supplemental Files for specific survey questions). Throughout the entire survey, all response fields included “I prefer not to say” as an opt-out choice. Once demographic information was collected from participants, the study’s full definition of an unprofessional peer review was presented and respondents self-identified whether they had ever received a peer review comment as first author that fit this definition. Survey respondents answering “no” or “I prefer not to say” were automatically redirected to the end of the survey, as no additional questions were necessary. Respondents answering “yes” were asked a series of follow-up questions designed to determine the nature of the unprofessional comments, the total number of scholarly publications to date and the number of independent times an unprofessional review was experienced. The perceived impact of the unprofessional reviews on the scientific aptitude, productivity and career advancement of each respondent was assessed using the following questions: (1) To what degree did the unprofessional peer review(s) make you doubt your scientific aptitude? (1–5) 1 = not at all, 5 = I fully doubted my scientific abilities; (2) To what degree do you feel that receiving the unprofessional review(s) limited your overall productivity? Please rate this from 1 to 5. 1 = not at all, 5 = greatly limited number of publications per year; (3) To what degree do you feel that receiving the unprofessional review(s) delayed career advancement? Please rate this from 1 to 5. 1 = not at all, 5 = greatly impacted/delayed career advancement. Finally, respondents were invited to provide direct quotes of unprofessional reviews received (although the respondents were told they could remove any personal or identifying information such as field of study, pronouns, etc.). Participants choosing to share peer review quotes were able to specify whether they gave permission for the quote to be shared/distributed. Explicit permission from each respondent was received to use and distribute all quotes displayed in Fig. 1. At the end of the survey, all respondents were required to certify that all the information provided was true and accurate to the best of their abilities and that the information was shared willingly. We recognize that there are limitations to our survey design. First, our survey was only administered in English. There also may have been non-response bias for individuals who did not experience negative comments during peer review even though any advertisement of this survey indicated that we sought input from anyone who has ever published a peer-reviewed study as first author. There could also be a temporal element, where authors who received comments more recently may have responded differently than those who received unprofessional reviews many years ago. Additionally, the order of questions was not randomized and participants were asked to complete demographic information before answering questions about their peer review experience, which may have primed respondents to select answers that were more in line with racial or gender-specific stereotypes (Steele & Ambady, 2006). In order to maintain the anonymity of our respondents, we did not ask for any bibliometric data from the authors. Given that our sample of respondents represented a diverse array of career stages, STEM fields, countries of residence and racial/ethnic identities, we do not believe that any of the above significantly limits the interpretation of our results. Data analysis We tested for the pervasiveness and downstream effects of unprofessional peer reviews on four intersecting gender and racial/ethnic groups: (1) women of color and non-binary people of color, (2) men of color, (3) white women and white non-binary people and (4) white men. Due to the small number of respondents identifying as non-binary (<1% of respondents), we statistically analyzed women and non-binary genders together in a category as marginalized genders in the sciences. However, refer to Table S1 for full breakdown of responses from each gender identity so that patterns may be assessed by readers without the constraints conferred by statistical assumptions and analyses. To protect the anonymity of individuals that may be personally identified based on the combination of their race/ethnicity and gender, respondents who identified as a race or ethnicity other than white, including those who checked multiple racial and/or ethnic categories were grouped together for the statistical analysis (Fig. S1). It is important to note that by grouping the respondents into four categories, the analysis captures only the broad patterns for intersectional groups and does not relay the unique experiences of each respondent, which should not be discounted. Survey respondents (N = 1,106) were given the opportunity to opt out of any question; therefore, the sample sizes were different for each statistical analysis. We tested for differences in the probability of receiving an unprofessional peer review across four intersectional groups (N = 1,077) using a Bayesian logistic regression (family = Bernoulli, link = logit). Of the 642 people who indicated that they received an unprofessional peer review, 617, 620 and 618 answered the questions regarding perceived impacts to their scientific aptitude, productivity and career advancement, respectively. We ran individual Bayesian ordinal logistic regressions (family = cumulative, link = logit) for each of the three questions to test for differences in probabilities of selecting a 1–5 across the four groups. All models were run using the BRMS package (Bürkner & Others, 2017) in Rv3.5.2 which uses the Hamiltonian Monte Carlo algorithm in STAN (Hoffman & Gelman, 2014; Stan Development Team, 2015). Each model was sampled from four chains, 4,000 iterations post-warmup, and half student t distributions for all priors. Model convergence was assessed using Gelman–Rubin diagnostics ( R ^ < 1 for all parameters; Gelman & Rubin, 1992) and visual inspection of trace plots. Posterior predictive checks were visually inspected using the pp_check() function and all assumptions were met. Data are presented as medians and two-tailed 95% Bayesian credible intervals (BCI).

Results We received 1,106 responses from people in 46 different countries across >14 STEM disciplines (Fig. 2). Overall, 58% of all the respondents (N = 642) indicated that they had received an unprofessional review, with 70% of those individuals reporting multiple instances (3.5 ± 5.8 reviews, mean ± SD, across all participants). There were no significant differences in the likelihood of receiving an unprofessional review among the intersectional groups (Fig. S2); however, there were clear and consistent differences in downstream effects between groups in perceived impacts on self-confidence, productivity and career trajectories after receiving an unprofessional review. Figure 2: Survey demographics. (A) Representative career stages (N = 11), (B) scientific disciplines (N = 14) and (C) countries (N = 46) from survey participants. Color in subset (C) represents number of surveys from each country where white is 0. White men were most likely to report no impact to their scientific aptitude (score of 1) after receiving an unprofessional peer review (P[1] = 0.40, 95% BCI [0.34–0.47], where P[score] denotes the probability of selecting a particular score), with a 5.7 times higher probability of selecting a 1 than a 5 (fully doubted their scientific aptitude; P[5] = 0.07, 95% BCI [0.05–0.09]). Notably, white men were 1.3, 2.0 and 1.7 times more likely to indicate no resultant doubt of their scientific aptitude than men of color (P[1] = 0.30, 95% BCI [0.20–0.41]), white women and white non-binary people (P[1] = 0.20, 95% BCI [0.16–0.23]) and women of color and non-binary people of color (P[1] = 0.23, 95% BCI [0.16–0.31]), respectively (Fig. 3A). Together, these results indicate that receiving unprofessional peer reviews had less of an overall impact on the scientific aptitude of white men relative to the remaining three groups. Figure 3: Results from Bayesian ordinal logistic regression. Figure shows the probability of selecting a 1–5 for (A) doubting scientific aptitude (N = 617), (B) delayed productivity (N = 620) and (C) delayed career advancement (N = 618) across intersectional groups after receiving an unprofessional peer review. Data are medians and two-tailed 95% BCI. Colors represent level of impact with the lightest (1) as no perceived impact and the darkest (5) as the highest impact. Women and non-binary people were grouped for the statistical analysis to represent marginalized genders in STEM fields. Similar patterns among intersectional groups emerged for reported impacts of unprofessional reviews on productivity (measured in number of publications per year). Specifically, women of color and non-binary people of color, white women and white non-binary people and men of color were mostly likely to select a 3 (moderate level of perceived negative impact on productivity), whereas white men were most likely to select a 1 (no perceived impact on their productivity; Fig. 3B). White men were also the least likely of all groups to indicate that receiving unprofessional reviews greatly limited their number of publications per year (P[5] = 0.06, 95% BCI [0.05–0.09]), which significantly differed from groups of women and non-binary people, but not men of color (Fig. 3B). Women of color and non-binary people of color had the most distinct pattern in reported negative impacts on career advancement (Fig. 3C). Women of color and non-binary people of color had a nearly equal probability of reporting each level of impact (1–5); whereas, men of color, white women and white non-binary people and white men had a decreasing probability of selecting scores indicative of a higher negative impact on career advancement (Fig. 3C). Specifically, women of color and non-binary people of color were the most likely to select that they had significant delays in career advancement as a result of receiving an unprofessional review (P[5] = 0.20, 95% BCI [0.13–0.28]). Women of color and non-binary people of color were also the least likely of the groups to report no impact on career advancement as a result of unprofessional reviews (P[1] = 0.22, 95% BCI [0.15–0.31]).

Conclusions Our study shows that unprofessional peer reviews are pervasive and that they disproportionately harm underrepresented groups in STEM. Specifically, underrepresented groups were most likely to report direct negative impacts on their scientific aptitude, productivity and career advancement after receiving an unprofessional peer review. While it was beyond the scope of this study, future investigations should also focus on the effect of unprofessional peer reviews on first-generation scientists English as a second language, career stage, peer review in grants, and other factors that could lead to differences in downstream effects. Unprofessional peer reviews have no place in the scientific process and individual scientists have the power and responsibility to enact immediate change. However, we recognize and applaud those reviewers and editors (and there are many!) that spend a significant amount of time and effort writing thoughtful, constructive, and detailed criticisms that are integral to moving science forward.