Future career impact

We therefore traced the publication records of PIs from the two groups. We first focus on active PIs in the NIH system, defined as those who apply for and/or receive NIH grants at some point in the future (Fig. 1c and ‘Different definitions of active PIs’ in Supplementary Note 3). We calculated the publication rates of the PIs, finding that the two groups published a similar number of papers per person over the next ten-year period (Fig. 2a), consistent with prior studies8,15,52. We then computed, out of the papers published by the near-miss and narrow-win group, the probability of finding hit papers (Fig. 2b), defined as being in the top 5% of citations received in the same year and field (as indicated by the Web of Science subject category)34,37. In the first five years, 13.3% of papers published by the narrow-win group turned out to be a hit paper, which is substantially higher than the baseline hit rate of 5%, demonstrating that narrow wins considered in our sample produced hit papers at a much higher rate than average scientists in their field. We measured the same probability for the near-miss group, finding that they produced hit papers at an average rate of 16.1%. Comparing the two groups, we find near misses outperformed narrow wins significantly, by a factor of 21% (χ2-test p-value < 0.001, odds ratio = 1.25). This performance advantage persisted: We analyzed papers produced in the second five-year window (6–10 years after treatment), uncovering a similar gap (Fig. 2b, χ2-test p-value < 0.001, odds ratio = 1.19). To ensure the observed effect is not just limited to hit papers, we also quantified performance using other commonly used measures, including average citations received within five years of publication (Fig. 2c) and the relative citation ratio (RCR) of each paper (see ‘Normalized citations over time and disciplines’ in Supplementary Note 3)53,54, arriving at the same conclusions. Indeed, papers published by the near-miss group in the next two five-year periods attracted on average 19.4% (32.3 for near misses and 27.0 for narrow wins, t-test p-value < 0.001, Cohen’s d = 0.08) and 12.0% more citations (32.3 for near misses and 28.8 for narrow wins, t-test p-value < 0.001, Cohen’s d = 0.06) than those by the narrow-win group, respectively (Fig. 2c).

Fig. 2 Comparing future career outcome between near misses (orange) and narrow wins (blue). a The average number of publications per person. b Near misses outperformed narrow wins in terms of the probability of producing hit papers in the next 1–5 years, 6–10 years, and 1–10 years. Note that there appears a slight performance improvement for the narrow-win group in the second five-year period, but the difference is not statistically significant (χ2-test p-value > 0.1, odds ratio = 1.05). c Average citations within 5 years of publication. The near-miss applicants again outperformed their narrow-win counterparts. To ensure all papers have at least 5 years to collect citations, here we used data from 1990 to 2000 to avoid any boundary effect. d Funding difference between the near-miss and narrow-win group from the NIH (near misses minus narrow wins). ***p < 0.001, **p < 0.05, *p < 0.1; Error bars represent the standard error of the mean Full size image

To further test the robustness of our results we repeated our analyses along several dimensions. We changed our definitions of junior PIs to two alternatives, by focusing on first-time R01 applicants only and by restricting to those without any current NIH grants (‘Alternative definitions of junior PIs’ in Supplementary Note 3). We varied our definitions of hit papers (from top 1% to top 15% of citations, ‘Varying thresholds for the definitions of hit papers’ in Supplementary Note 3). We computed per capita measures of hit papers (‘Hits per capita’ in Supplementary Note 3). We adjusted for field differences of citations, by calculating the average normalized citations by field and year34 (‘Normalized citations over time and disciplines’ in Supplementary Note 3). We also varied our definition of fields using the Medical Subject Headings (MeSH)55 (‘Different field definition’ in Supplementary Note 3). We repeated our analyses across different measurement time periods (‘Robustness to alternative fiscal years’ in Supplementary Note 3). We also checked whether the results may be affected by pre-existing papers moving through the publication process (‘Publication lags’ in Supplementary Note 3). We further repeated our analyses by controlling ex post funding status for narrow wins and near misses (‘Robustness for ex post funding status’ in Supplementary Note 3). We also tried several name disambiguation methods and repeated our analyses (‘Author name disambiguation’ in Supplementary Note 1). Amid all variations, the conclusions remain the same.

The performance advantage of the near misses is particularly surprising given that the narrow wins, by construction, had an initial NIH funding advantage immediately after treatment. Given that funding from the NIH can be an important means to augmenting scientific production35,38,39,52,56,57,58, we investigate funding dynamics for the near-miss and narrow-win groups over the following ten-year period. We find that the near-miss group naturally received significantly less NIH funding in the first five years following treatment, averaging $0.29 million less per person (Fig. 2d, t-test p-value < 0.05, Cohen’s d = 0.28), which is consistent with prior studies8,52,59. Yet the funding difference between the two groups disappeared in the second five-year period (Fig. 2d, t-test p-value > 0.1, Cohen’s d = 0.02). Although the NIH is the world’s largest funder for biomedical research, near misses might have obtained more funding elsewhere (see ‘Additional funding by near misses’ in Supplementary Note 3 for details). To test this hypothesis, we further collected individual grant histories for PIs in our sample from the Dimensions data, allowing us to calculate the total funding support from agencies worldwide beyond NIH. We first measured the total funding support from the U.S. National Science Foundation (NSF) received by individuals with the same name in the same period, finding narrow wins obtained significantly more NSF funding within 5 years after treatment. We further calculated the total funding support from agencies other than the NIH or NSF, finding that near misses did not acquire more funding than narrow wins. We also manually checked acknowledgment statements within a fraction of papers published by the two groups, finding again the same conclusion.

Together, these results demonstrate that over the course of ten years, near misses had fewer initial grants from the NIH and NSF. Yet they ultimately published as many papers and, most surprisingly, produced work that garnered substantially higher impacts than their narrow-win counterparts.

Is the uncovered difference in outcomes causally attributable to the early-career setback? Or, could it be explained by other alternative forces? Indeed, there might still exist observable or otherwise unobserved factors that affect funding success near the threshold (e.g., individual characteristics60, fields of study, personality traits, etc.), which might also drive future career outcomes. To rule out alternative explanations, we leverage two additional inference techniques, Coarsened Exact Matching (CEM)61,62 and fuzzy Regression Discontinuity (RD)63,64. We first matched near misses and narrow wins with respect to a wide range of observable characteristics (see Methods section for further description of CEM), and find that after matching, near misses still outperformed narrow wins in terms of both hit papers (16.4% for near misses, 14.0% for narrow wins, χ2-test p-value < 0.001, odds ratio = 1.20) and average citations per paper (30.8 for near misses and 27.7 for narrow wins, t-test p-value < 0.001, Cohen’s d = 0.05, see ‘Matching strategy and additional results in the RD regression’ in Supplementary Note 3 for details). While matching can only eliminate potential observable features, we further mitigate the effect of other observable and unobservable influences using the RD analysis. Specifically, we use an indicator for the score being above or below the funding threshold as an instrumental variable (IV), rather than the actual funding outcome itself, to predict future career outcomes (see Methods section). The RD approach helps us rule out unobserved influences on funding outcome or any otherwise unobserved individual characteristics that differ smoothly with the score63,64, allowing us to further establish a causal link between early-career near miss and future career impact. By accounting for any potential confounding factors, our RD estimates indicate that one early-career near miss increases the probability of publishing a hit paper in the next 10 years by 6.1% (p-value = 0.041), and the average citations per paper by 34% (9.67 citations in 5 years, p-value = 0.046) (see Methods section). The RD analyses help establish the causal interpretation of our results, and the agreement in results across all the methods further demonstrates the robustness of our findings.

These results document that, despite an early setback, near misses outperformed narrow wins over the longer run, conditional on remaining active in the NIH system. This finding itself has a striking implication. Indeed, take two researchers who are seeking to continue their careers in science. While both near-miss and narrow-win applicants published high-impact papers at a higher rate than their contemporary peers, comparing between the two groups, it is the ones who failed that are more likely to write a high-impact paper in the future.

To conceptualize this finding, consider two hypotheses. The first is a screening hypothesis, where the population of survivors among the near-miss group may have fixed, advantageous characteristics. Second, the result is consistent with failure itself teaching valuable lessons or strengthening resolve. To help unpack the findings, we examine differential survival rates between two samples and further ask whether the screening hypothesis alone may be sufficient to explain the observed difference in outcomes.

Screening hypothesis

We first investigate attrition rates by studying the percentage of the initial PIs who remained active in the NIH system and find that the attrition rate of the two groups differed significantly (Fig. 3a). In the year immediately following treatment, the near-miss group had 11.2% fewer active PIs than the narrow-win group (χ2-test, p-value < 0.001). This difference is not simply because narrow wins received an initial grant. Indeed, the gap persisted and extended beyond the first five years, remaining at 11.8% in year seven (χ2-test, p-value = 0.002), followed by a drop afterwards. The RD analysis indicates that an early-career near miss on average led to a 12.6% chance of disappearing permanently from the NIH system over the next ten years (see Methods section). These results thus highlight the fragility of a junior scientific career, with one early near miss being associated with significantly higher attrition from the NIH system, despite the fact that to become an NIH PI, one had to go through years of training with a demonstrated track record of research. Notwithstanding the evidence that PhDs who left science are disproportionally employed at large, high-wage establishments65, Fig. 3a documents differential survivorship between narrow wins and near misses, which raises the important next question: Could screening alone account for the observed performance advantage?

Fig. 3 Testing the screening hypothesis with a conservative removal procedure. a Attrition rate difference between the near-miss and narrow-win group (near misses minus narrow wins). We measure the percentage of PIs remained in each of the two groups, and calculate their difference in each of the ten years after treatment. b An illustration of the conservative removal procedure. To test if the observed performance difference can be accounted for by the population difference, we performed a conservative estimation by removing PIs who published the fewest hit papers but with the most publications from the narrow-win group (blue), such that after removal (green) the two groups have the same fractions of PIs remaining. After removal, the near-miss group still outperformed the narrow-win group in terms of the probability of producing a hit paper (χ2 test p-value < 0.001, odds ratio = 1.17) (c), or the average citations of papers (t-test p-value < 0.001, Cohen’s d = 0.06) (d). The results shown in c–d suggest that while the performance of narrow wins indeed improved following the conservative removal procedure, the screening hypothesis alone cannot account for the uncovered performance gap. ***p < 0.001, **p < 0.05, *p < 0.1; Error bars represent the standard error of the mean Full size image

To understand the nature of the potential screening effect, we first test its underlying assumption by comparing pre-treatment characteristics of near misses and narrow wins who remained ex post, finding a lack of difference between these two groups in any observable dimension ex ante (Supplementary Fig. 29a), which suggests the screening effect, if any, may be modest (‘On the screening mechanism’ in Supplementary Note 3). To further examine potential screening effects, we removed PIs from narrow wins, such that the attrition rate following removal is the same between the two groups (Fig. 3b). We performed a conservative estimation by removing PIs from narrow wins who, ex post, published the fewest hit papers but had the most publications. In other words, we created a subpopulation of narrow wins that had the same attrition rate as the near misses but are aided by an artificial upward adjustment to their hit probabilities (‘On the screening mechanism’ in Supplementary Note 3). We find that, while the performance of narrow wins improves by construction following this conservative removal procedure, the improvement is not sufficient to account for the observed performance gap. Indeed, in terms of the probability of producing a hit paper, or the average citations per paper, near misses still outperformed narrow wins (Fig. 3c, d). The matching and the RD yield consistent conclusions (‘Matching strategy and additional results in the RD regression’ in Supplementary Note 3). Together, these results demonstrate that the screening effect may have played a role, but it appears insufficient to entirely account for the observed difference between near misses and narrow wins.

We clarify these results further on several dimensions. To understand if the average improvement of the near misses masks heterogeneous responses, we measured the coefficient of variation for citations, finding a lack of difference between the two groups, suggesting a homogeneous improvement within the group (‘Variance and outliers’ in Supplementary Note 3, Supplementary Fig. 23). We also compared the median citations to eliminate the role of outliers, yielding the same conclusion (‘Variance and outliers’ in Supplementary Note 3). To rule out collaboration effects, whereby early-career setbacks might lead junior scientists to seek out advantageous collaborations, we restricted our analyses to lead-author publications only, and controlled for the status of their collaborators, yielding the same conclusions (Supplementary Figs. 13, 27). Further, to check that the uncovered performance gap is not simply because narrow wins became worse, we selected a group of clear winners whose scores were further removed from the funding threshold. We find, as expected, that this group of PIs performed substantially better than the near-miss group prior to treatment. Yet, in the ten years afterwards, they show a similar performance as the near-miss group (‘Was it because narrow wins became worse?’ in Supplementary Note 3), indicating that near misses performed at a comparable level as the group that appeared demonstrably better than them initially. To test the hypothesis that narrow wins were committed to initially proposed ideas, we compared articles by narrow wins published in 5 years after treatment with those published between 6 to 10 years. We find no statistically significant improvement for narrow wins in terms of probability to publish hit papers (χ2-test p-value > 0.1) or normalized citations (t-test p-value > 0.1) (‘Was it because narrow wins became worse?’ in Supplementary Note 3). We also controlled for fixed effects categorizing PIs’ prior NIH experience, recovering the same conclusions (Supplementary Fig. 31). We also repeated all our analyses by varying our definition of active PIs by focusing on publishing scientists only (‘Different definitions of active PIs’ in Supplementary Note 3, Supplementary Fig. 25), and the definition of pay lines by using the NIH percentile score instead of priority score (Supplementary Fig. 26). Amid all variations, our findings remain the same.

Beyond citations

While citations and their variants have been used extensively to quantify career outcomes7,41,45,66,67,68,69, they may represent an imperfect or limited proxy for measuring output, prompting us to ask if the observed effect of early-career setback extends beyond citation measures. To this end, we used additional datasets to calculate three indicators probing the clinical relevance of the works. These measures are: (1) whether a paper is a clinical trial publication (direct contribution to clinical translation); (2) whether a paper has been cited by at least one clinical trial publication (indirect contribution to clinical translation), and (3) whether a paper has potential to become translational research70 (potential for translation). We compared works produced by near misses and narrow wins over the ten-year period, finding that across all three translational dimensions, near misses systematically outperformed narrow wins. Specifically, near misses were 50% more likely to publish a clinical trial paper compared with narrow wins (4.8% for near misses, 3.2% for narrow wins, χ2-test p-value < 0.001, odds ratio = 1.53, Fig. 4a), and their overall publications were 19.6% more likely to be cited by clinical trials (34% for near misses, 28.4% for narrow wins, χ2-test p-value < 0.001, odds ratio = 1.30, Fig. 4b), and are 24.5% higher in their potential for bench-to-bedside translation (35.4% for near misses, 28.4% for narrow wins, χ2-test p-value < 0.001, odds ratio = 1.38, Fig. 4c). We also find that, all these conclusions remain the same after conducting the conservative removal procedure as described in Fig. 3 (Fig. 4e–g). Finally, to test if the tendency toward clinical research can by itself account for the observed citation difference between near misses and narrow wins, we separated their publications into clinical and non-clinical papers, finding that within non-clinical papers, near misses again outperformed narrow wins (Fig. 4d, h). We repeated all our analyses using the RD approach, recovering broadly consistent conclusions (‘Matching strategy and additional results in the RD regression’ in Supplementary Note 3). Together, the results shown in Fig. 4 suggest that the uncovered effect of early-career setback goes beyond citation measures, with near misses outperforming narrow wins in both basic and translational science.