Assessment of researchers is necessary for decisions of hiring, promotion, and tenure. A burgeoning number of scientific leaders believe the current system of faculty incentives and rewards is misaligned with the needs of society and disconnected from the evidence about the causes of the reproducibility crisis and suboptimal quality of the scientific publication record. To address this issue, particularly for the clinical and life sciences, we convened a 22-member expert panel workshop in Washington, DC, in January 2017. Twenty-two academic leaders, funders, and scientists participated in the meeting. As background for the meeting, we completed a selective literature review of 22 key documents critiquing the current incentive system. From each document, we extracted how the authors perceived the problems of assessing science and scientists, the unintended consequences of maintaining the status quo for assessing scientists, and details of their proposed solutions. The resulting table was used as a seed for participant discussion. This resulted in six principles for assessing scientists and associated research and policy implications. We hope the content of this paper will serve as a basis for establishing best practices and redesigning the current approaches to assessing scientists by the many players involved in that process.

Abbreviations: ACUMEN, Academic Careers Understood through Measurement and Norms; CSTS, Centre for Science and Technology Studies; CV, curriculum vita; DOI, digital object identifier; DORA, Declaration on Research Assessment; GBP, Great Britain Pound; GEP, Good Evaluation Practices; ISNI, International Standard Name Identifier; JIF, journal impact factor; METRICS, Meta-Research Innovation Center at Stanford; NAS, National Academy of Sciences; NIH, National Institutes of Health; NIHR, National Institute for Health Research; OA, open access; PQRST, Productive, high-Quality, Reproducible, Shareable, and Translatable; RCR, relative citation ratio; REF, Research Excellence Framework; REWARD, Reduce research Waste And Reward Diligence; RIAS, responsible indicator for assessing scientists; SNIP, Source Normalized Impact per Paper; SWAN, Scientific Women's Academic Network; TOP, transparency and openness promotion; UMC, Utrecht Medical Centre

Copyright: © 2018 Moher et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Using more appropriate incentives and rewards may help improve clinical and life sciences and their impact at all levels, including their societal value. We set out to ascertain what has been proposed to improve the evaluation of ‘life and clinical’ research scientists, how a broad spectrum of stakeholders view the strengths and weaknesses of these proposals, and what new ways of assessing scientists should be considered.

Few assessments of scientists focus on the use of good or bad research practices, nor do currently used measures tell us much about what researchers contribute to society—the ultimate goal of most applied research. In applied and life sciences, the reproducibility of findings by others or the productivity of a research finding is rarely systematically evaluated, in spite of documented problems with the published scientific record [ 11 ] and reproducibility across all scientific domains [ 12 – 13 ]. This is compounded by incomplete reporting and suboptimal transparency [ 14 ]. Too much research goes unpublished or is unavailable to interested parties [ 15 ].

That faculty hiring and advancement at top institutions requires papers published in journals with the highest JIF (e.g., Nature, Science, Cell) is more than just a myth circulating among postdoctoral students [ 3 – 6 ]. Emphasis on JIF does not make sense when only 10%–20% of the papers published in a journal are responsible for 80%–90% of a journal’s impact factor [ 7 , 8 ]. More importantly, other aspects of research impact and quality, for which automated indices are not available, are ignored. For example, faculty practices that make a university and its research more open and available through data sharing or education could feed into researcher assessments [ 9 , 10 ].

Assessing researchers is a focal point of decisions about their hiring, promotion, and tenure. Building, writing, presenting, evaluating, prioritising, and selecting curriculum vitae (CVs) is a prolific and often time-consuming industry for grant applicants, faculty candidates, and assessment committees. Institutions need to make decisions in a constrained environment (e.g., limited time and budgets). Many assessment efforts assess primarily what is easily determined, such as the number and amount of funded grants and the number and citations of published papers. Even for readily measurable aspects, though, the criteria used for assessment and decisions vary across settings and institutions and are not necessarily applied consistently, even within the same institution. Moreover, several institutions use metrics that are well known to be problematic [ 1 ]. For example, there is a large literature on the problems with and alternatives to the journal impact factor (JIF) for appraising citation impact. Many institutions still use it to assess faculty through the quality of the literature they publish in, or even to determine monetary rewards [ 2 ].

Table 1 served as the basis for an initial discussion about the problems of assessing scientists. This was followed by discussions of possible solutions to the problems, new approaches for promotion and tenure committees, and implementation strategies. All discussions were recorded, transcribed, and read by five coauthors. For this, six general principles were derived. This summary was then shared with all meeting participants for additional input.

We invited 23 academic leaders: deans of medicine (e.g., Oxford), public and foundation funders (e.g., National Institutes of Health [NIH]), health policy organisations (e.g., Science Policy, European Commission; Belgium), and individual scientists from several countries. They were selected based on contributions to the literature on the topic and representation of important interests and constituencies. Twenty-two were able to participate (see S1 Table for a complete list of participants and their affiliations). Prior to the meeting, all participants were sent the results of a selected review of the literature distilled into a table (see Table 1 ) and several selected readings.

Broad criteria were used for article inclusion (the article focus had to be either bibliometrics, research evaluation, and/or management of scientists and it had to be reported in English). Two of us selected the potential papers and at least three of us reviewed and discussed each selection for its inclusion. From each included article we extracted the following information: authors, name of article/report and its geographic location, the authors’ stated perspective of the problem assessing research and scientists, the authors’ description of the unintended consequences of maintaining the current assessment scheme, the article’s proposed solutions, and our interpretation of the potential limitations of the proposal. The resulting table (early version of Table 1 ) along with a few specific publications was shared with the participants in advance of the meeting in the hopes that it would stimulate thinking on the topic and be a reference source for discussions.

To help address this goal, we convened a 1-day expert panel workshop in Washington, DC, in January 2017. Pre-existing commentaries and proposals to assess scientists were identified with snowball techniques [ 16 ] (i.e., an iterative process of selecting articles; the process is often started with a small number of articles that meet inclusion criteria; see below) to examine the literature to ascertain what other groups are writing, proposing, or implementing in terms of how to assess scientists. We also searched the Meta-Research Innovation Center at Stanford (METRICS) research digest and reached out to content experts. Formal searching proved difficult (e.g., exp*Reward/(7408); reward*.ti,kw (9708), incentiv*.ti,kw. (5078)) and resulted in a very large number of records with low sensitivity and recall. We did not set out to conduct a formal systematic review of every article on the topic.

Results

We included a list of 21 documents [11,17–36] in Table 1. There has been a burgeoning interest in assessing scientists in the last 5 years (publication year range: 2012–January 2017). An almost equal number of documents originated from the US and Europe (one also jointly from Canada). We divided the documents into four categories: large group efforts (e.g., the Leiden Manifesto [20]), smaller group or individual efforts (e.g., Ioannidis and Khoury’s Productive, high-Quality, Reproducible, Shareable, and Translatable [PQRST] proposal [29]); journal activities (e.g., Nature [32]); and newer quantitative metric proposals (e.g., journal citation distributions [34]).

We interpreted all of the documents to describe the problems of assessing science and scientists in a similar manner. There is a misalignment between the current problems in research. The right questions are not being asked; the research is not appropriately planned and conducted; and when the research is completed, results remain unavailable, unpublished, or get selectively reported; reproducibility is lacking as is evidence about how scientists are incentivised and rewarded. We interpreted that several of the documents pointed to a disconnect between the production of research and the needs of society (i.e., productivity may lack translational impact and societal added value). We paraphrased the views expressed across several of the documents: ‘we should be able to improve research if we reward scientists specifically for adopting behaviours that are known to improve research’. Finally, many of the documents described the JIF as an inadequate measure for assessing the impact of scientists [19,20,29]. The JIF is commonly used by academic institutions [4,5,37] to assess scientists, although there are efforts to encourage them not to do so. Not modifying the current assessment system will likely result in the continued bandwagon behaviour that has not always resulted in positive societal behaviour [25,38]. Acting now to consider modifying the current assessment system might be seen as a progressive move by current scientific leaders to improve how subsequent generations of scientists are evaluated.

Large group proposals for assessing scientists Nine large group efforts were included [11,17–24] (see Table 1), representing different stakeholder groups. The Leiden Manifesto [20] and the Declaration on Research Assessment (DORA) [19] were both developed at academic society meetings and are international in their focus, whereas the Metric Tide [21] was commissioned by the UK government (operating independently) and is more focused on the UK academic marketplace. The Leiden Manifesto authors felt that research evaluation should positively support the development of science and its interactions with society. It proposes 10 best practices, for example, that expert qualitative assessment should take precedence, supported by the quantitative evaluation of a researcher using multiple indices. Several universities have recently pledged to adopt these practices [39]. The San Francisco DORA, which is also gaining momentum [40–42], was first developed by editors and publishers and focuses almost exclusively on the misuse of the JIF. DORA describes 17 specific practices to diminish JIF dependence by four stakeholder groups: scientists, funders, research institutions, and publishers. DORA recommends focusing on the content of researchers’ output, citation of the primary literature, and using a variety of metrics to show the impact. Within evidence-based medicine, systematic reviews are considered stronger evidence than individual studies. DORA’s clarification about citations and systematic reviews might facilitate its endorsement and implementation within faculties of medicine. The National Academy of Sciences proposed that scientists should be assessed for the impact of their work rather than the quantity of it [22]. Research impact is also raised as an important new assessment criterion in the UK’s Research Excellence Framework (REF) [24], an assessment protocol in which UK higher education institutions and their faculty are asked to rate themselves on three domains (outputs, impact, and environment) across 36 disciplines. These assessments are linked to approximately a billion Great Britain Pounds (GBPs) of annual funding to universities, of which 20% (to be increased to 25% in the next round) is based on the impact of the faculty member’s research. The inclusion of assessing impact (e.g., through case studies) has fostered considerable discussion across the UK research community [43]. Some argue it is too expensive, that pitting universities against each other can diminish cooperation and collegiality, that it does not promote innovation, and that it is redundant [44]. The Metric Tide, which has been influential in the UK as it relates to that country’s REF, made 20 recommendations related to how scientists should be assessed [21]. It recommends that universities should be transparent about how faculty assessment is performed and the role that bibliometrics plays in such assessment, a common theme across many of the efforts (e.g., [19,20]).

Individual or small group proposals for assessing scientists Six smaller group or individual proposals were included [25–30] (see Table 1). For example, Mazumdar and colleagues discussed the importance of rewarding biostatisticians for their contributions to team science [28]. They proposed a framework that separately assesses biostatisticians for their unique scientific contributions, such as the design of a grant application and/or research protocol, and teaching and service contributions, including mentoring in the grant application process. Ioannidis and Khoury have proposed scientist assessments revolving around ‘PQRST’: Productivity, Quality, Reproducibility, Sharing, and Translation of research [29]. Benedictus and Miedema describe approaches currently being used at the Utrecht Medical Centre, the Netherlands [25]. For the Utrecht assessment system, a set of indicators was defined and introduced to evaluate research programs and teams that mix the classical past-performance bibliometric measures with process indicators. These indicators include evaluation of leadership, culture, teamwork, good behaviours and citizenship, and interactions with societal stakeholders in defining research questions, the experimental approaches, and the evaluation of the ongoing research and its results. The latter includes semi-hybrid review procedures, including peers and stakeholders from outside academia. The new assessment for individual scientists is complemented by a semi-qualitative assessment in a similar vein for our multidisciplinary research programs. Taking a science policy perspective, these institutional policies are currently being evaluated with the Centre for Science and Technology Studies (CSTS) in Leiden by investigating how evaluation practices reshape the practice of academic knowledge production. Not surprisingly, many of these proposals and individual efforts overlap. For example, the Leiden Group’s fourth recommendation, ‘Keep data collection and analytical processes open, transparent and simple’, is similar to DORA’s 11th specific (of 17) recommendation, ‘Be open and transparent by providing data and methods used to calculate all metrics’. Some groups use assessment tools as part of their solutions. The Academic Careers Understood through Measurement and Norms (ACUMEN) group produced a weblike document system [17], whereas the Mazumdar group created checklists [28]. Groups targeted different groups of stakeholders. The Nuffield Council on Bioethics targeted funders, research institutions, publishers and editors, scientists, and learned societies and professional bodies [23], as did the Reduce research Waste And Reward Diligence (REWARD) Alliance, who added regulators as one of their target groups [11]. Most of the proposals were aspirational in that they were silent on details of how exactly faculty assessment committees could implement their proposals, what barriers to implementation existed, and how to overcome them. Integrating behavioural scientists and implementation scientists into implementation discussions would be helpful. Ironically, most of the proposals do not discuss whether or how to evaluate the impact of adopting them.

Journal proposals for assessing scientists Journals may not appear to be the most obvious group to weigh in on reducing the reliance on bibliometrics to assess scientists. Traditionally, they have been focused on (even obsessed with) promoting their JIFs. Yet, some journals are beginning to acknowledge the JIF’s possible limitations. For example, the PLOS journals do not promote their JIFs [45]. BioMed Central and Springer Open have signed DORA, stating, ‘In signing DORA we support improvements in the ways in which the output of scientific research is evaluated and have pledged to “greatly reduce emphasis on the journal Impact Factor as a promotional tool by presenting the metric in the context of a variety of journal-based metrics”’ [46]. We included two journal proposals [31,32]. Nature has proposed, and in some cases implemented, a broadening of how they use bibliometrics to promote their journals [32]. We believe a reasonable short-term objective is for journals to provide more potentially credible ways for scientists to use journal information in their assessment portfolio. eLife has proposed a menu of alternative metrics that go beyond the JIF, such as social and print media impact metrics, that can be used to complement article and scientist assessments [31]. Journals can also be an instrument for promoting best publication practices among scientists, such as data sharing, and academic institutions can focus on rewarding scientists for employing those practices rather than quantity of publications alone. Reporting biases, including publication bias and selective outcome reporting, are prevalent [14]. A few journals, particularly in the psychological sciences, have started using a digital badge system that promotes data sharing, although there has been some criticism of them [47]. The journal Psychological Science has evaluated whether digital badges result in more data sharing [48]. Such badges may potentially be used for assessing scientists based on whether they have adhered to these good publication practices. As of mid-2017, 52 mostly psychology journals have agreed to review and accept a paper at the protocol stage if a ‘registered report’ has been recorded in a dedicated registry [49]. If such initiatives are successful, assessors could reward scientists for registering their protocols, as with clinical trials and systematic reviews they can currently monitor timely registration in one of several dedicated registries.

Proposals to improve quantitative metrics There are many efforts to improve quantitative metrics in the vast and rapidly expanding field of scientometrics. We included four proposals [33–36]. An influential group including editors (Nature, eLife, Science, EMBO, PLOS, The Royal Society), a scientometrician, and an advocate for open science proposed that journals use the journal citation distribution instead of the JIF [36]. This allows readers to examine the skewness and variability in citations of published articles. Similarly, the relative citation ratio (RCR) or the Source Normalized Impact per Paper (SNIP) have been proposed to adjust the citation metric by content field, addressing one of the JIF’s deficiencies [19]. Several proposals have recognised the need for field-specific criteria [20,21]. However, the value of different normalisations needs further study [50] and there has been criticism of the proposed RCR [51]. There are currently dozens of citation indicators that can be used alone or in combination [52]. Each has its strengths and weaknesses, including the possibility for ‘gaming’ (i.e., manipulation by the investigator). Besides citation metrics, there is also increasing interest in developing standardised indicators for other features of impact or research practices, for example, alternative metrics, as discussed above. For example, Twitter, Facebook, or lay press discussions might indicate influence on or accessibility to patients. However, alternative metrics can also be gamed and social media popularity may not correlate with scientific, clinical, and/or societal benefit. An ‘S-index’ has been proposed to assess data sharing [36], albeit with limitations (see Table 1).

Principles for assessing scientists Six general principles emerged from the discussions, each with research and policy implications (see Table 2). Several have been proposed previously [53]. PPT PowerPoint slide

PowerPoint slide PNG larger image

larger image TIFF original image Download: Table 2. Key principles, participant dialogue, and research and policy implications when assessing scientists. https://doi.org/10.1371/journal.pbio.2004089.t002 The first principle is that contributing to societal needs is an important goal of scholarship. Focusing on research that addresses the societal need and impact of research requires a broader, outward view of scientific investigation. The principle is based on academic institutions in society, how they view scholarship in the 21st century, the relevance of patients and the public, and social action [10]. If promotion and tenure committees do not reward these behaviours, or penalise practices that diminish the social benefit of research, maximal fulfillment of this goal is unlikely [25]. The second principle is that assessing scientists should be based on evidence and indicators that can incentivise best publication practices. Several new ‘responsible indicators for assessing scientists’ (RIAS’s) were proposed and discussed. These include assessing registration (including registered reports); sharing results of research; reproducible research reporting; contributions to peer review; alternative metrics (e.g., uptake of research by social media and print media) assessed by several providers, such as Altmetric.com; and sharing of datasets and software assessed through Impact Story [54]). Such indicators should be measured objectively and accurately, as publication and citation tools do currently. Some assessment items, such as reference letters from colleagues and stakeholders affected by the research, cannot be converted into objective measurements, but one may still formally investigate their value [55]. As with any new measures, RIAS characteristics need to be studied in terms of ease of collection, their frequencies and distributions in different fields and institutions, the kind of systems needed to implement them, and their usefulness in both evaluation and modifying researcher behaviours and the extent to which each may be gamed. Different institutions could and should experiment with different sets of RIAS’s to assess their feasibility and utility. Ultimately, if there were enough consensus around a core set, institutional research funding could be tied to their collection, such as underlies successful implementation of Athena Scientific Women's Academic Network (SWAN) for advancing gender equity, which has been highly successful in the UK [56]. One barrier to implementation of any RIAS scheme is whether it would affect current university rankings (e.g., Times Higher Education World University Rankings). Productivity, measured in terms of publication output, is an important input into such rankings. Participants felt that any RIAS dashboard could be included in or as an alternative to university ranking schemes. However, these ranking systems are themselves problematic; the Leiden CSTS has recently proposed 10 principles regarding the responsible use of such ranking systems [57]. The third principle is that all research should be published completely and transparently, regardless of the results. Academic institutions could implement policies in the promotion process to review complete reporting of all research, and/or penalise noncompleted or nonpublished research—particularly clinical trials, which must be registered. For nonclinical research, participants discussed the need to reward other types of openness, such as sharing of datasets, materials, software and methods used, and explicit acknowledgment of their exploratory nature, when appropriate [58]. Finally, finding fair ways to reward team endeavors is critical, given the growing collaborative nature of research, which bibliometrics cannot properly assess. For example, some promotion and tenure committees largely disregard work for which the faculty candidate is not the first or senior author [4]. Conversely, citation metrics that do not correct for multiple coauthorship and thus authors who are just appearing in long author mastheads can result in inappropriately high citation metrics. The fourth principle relates to openness—facilitating dissemination and use of research data and results by others. Researchers can share their data, procedures, and code in various ways, such as in open access repositories. Some journals are supporting this process by endorsing and implementing the transparency and openness promotion (TOP) guidelines [59]. Groups that rank universities can also support this principle by sharing the underlying data used to make their assessments. The fifth principle requires investing in research to provide the necessary evidence to guide the development of new assessment criteria and to evaluate the merits of existing ones. Funders are the ones to make such investments and some, particularly in Europe (e.g., Netherlands Organisation for Scientific Research), have already started. The final principle involves rewarding researchers for intellectual risk-taking that might not be reflected in early successes or publications. The need for a young researcher to obtain their own funding early often results in a conservatism that is inimical to groundbreaking work at a time when they might be the most creative. Changing assessments to evaluate and reward such hypotheses might encourage truly creative research. It is also possible to conduct some forms of research with limited funding [60].

Implementation A challenge introducing any of these principles, or other new ideas, is how best to operationalise them. The TrialsTracker tool [61] enables institutions from around the world (with more than 30 trials) to monitor their trial reporting. Although the tool has limitations [62], it has a low barrier to implementation and provides a useful and easy starting point for audit and feedback. Promotion and tenure committees could receive such data as part of annual faculty assessment. They could also ask scientists to modify their CVs to incorporate information about registration through indicating the name and registration number of the registry, whether they have participated in a journal’s registered reports program, and a citation of the completed and published study. For each new initiative, it is important to generate evidence, ideally from experimental studies, on whether it leads to better outcomes. Participants also discussed that efforts to reward good, rigorously conducted evaluation should not come at the expense of stifling creative ‘blue-sky’ research primarily aimed at understanding biologic processes.