Abstract Several uncorroborated, false, or misinterpreted conceptions have for years been widely distributed in academic publications, thus becoming scientific myths. How can such misconceptions persist and proliferate within the inimical environment of academic criticism? Examining 613 articles we demonstrate that the reception of three myth-exposing publications is skewed by an ‘affirmative citation bias’: The vast majority of articles citing the critical article will affirm the idea criticized. 468 affirmed the myth, 105 were neutral, while 40 took a negative stance. Once misconceptions proliferate wide and long enough, criticizing them not only becomes increasingly difficult, efforts may even contribute to the continued spreading of the myths.

Citation: Letrud K, Hernes S (2019) Affirmative citation bias in scientific myth debunking: A three-in-one case study. PLoS ONE 14(9): e0222213. https://doi.org/10.1371/journal.pone.0222213 Editor: Lutz Bornmann, Max Planck Society, GERMANY Received: May 20, 2019; Accepted: August 23, 2019; Published: September 9, 2019 Copyright: © 2019 Letrud, Hernes. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability: All relevant data are contained in the paper and its Supporting Information file. Funding: The author(s) received no specific funding for this work. Competing interests: The authors have declared that no competing interests exist.

Introduction Some misconceptions become engrained in academic publishing and debates. Examples include the low risk of addiction from opioids prescribed for chronic pain [1], the ‘Patient Zero’ supposedly responsible for the U.S. AIDS epidemic [2], the Yerkes-Dodson law [3, 4], the endless behavioral loops of the digger sphex [5], the Learning Styles [6, 7], the Learning Pyramid models [8, 9], and the Hawthorne Effect [10]. Despite fundamental flaws, these claims have proliferated in academic publications for decades, some of them for more than a century. Even though the number of positively identified myths appears to be limited, there is reason to suspect that scientific myths such as these are not a marginal phenomenon. Several unwarranted claims have become part of the scientific corpus [11, 12], and these claims could potentially become entrenched as common knowledge in the way exemplified by the above myths. Citation bias [13–15] contribute to the academic cementation of ideas by favoring studies with positive evidence to those with negative evidence. Active efforts at countering scientific myth proliferation are required, lest they misinform descriptive and normative deliberations, and clutter encyclopedias, review articles, and topic searches in databases. However, once claims such as these become entrenched in academic discourses, efforts at criticizing them are counteracted by an affirmative citation bias: We theorize that there are three main ways of citing a critical paper (apart from remaining neutral). Consider the following scenario: A paper sets out to challenge a flawed yet widely distributed theory, and makes a well-argued case against it. If, on the one hand, readers accept the critical arguments and reject the theory, their citation will presumably reiterate the critique. On the other hand, those who disagree with the paper will cite it and make their case for why they choose to sustain the theory, engaging it in a debate. While a third group of readers will cite the critical paper as corroborating the theory, an instance of what Greenberg [15] terms ‘citation diversion’, presumably because they have not read it, or failed to understand it [16–21]. If these three groups are of equal size, those that uphold the model criticized will outnumber those who echoes the critique. Searching for evidence of affirmative citation bias, we perform case studies of the academic reception of three articles critical of the widely cited yet contentious Hawthorne Effect. By consulting papers citing these critical works, we seek to establish whether, and to what degree the accumulated citations are indeed skewed in favor of the Hawthorne Effect, suggesting the existence of an affirmative citation bias. We shall also search for evidence of citation diversion. The Hawthorne effect The idea of a Hawthorne Effect originated from studies on workplace behavior at the Western Electric Company’s Hawthorne Plant during the 1920s and 1930s, and was primarily based on studies of the relay assembly room, and on informal studies about worker response to changes of lighting [22, 23]. Surprisingly, both higher and lower lighting levels supposedly led to increased productivity. In 1941 Roethlisberger described the altered behavior of workers knowing that they are being observed [24], and in 1953 John French introduced the term ‘Hawthorne Effect’, describing an increase in productivity due to social position and social treatment [22]. ‘The Hawthorne Effect’ is now an ambiguous and vague, yet widely used, term, primarily associated with an observer effect: subjects altering their behavior when aware of being observed [25, 26]. The Hawthorne studies has been in the receiving end of extensive, and sometimes harsh, criticism [24, 27–29], as has the Hawthorne Effect [10, 22, 30].

Method We based the case studies on articles arguing against the Hawthorne Effect, interpreted as various observer effects. The selection criteria being that they were unequivocally critical of the effect, that their argumentation was substantial, and that they were extensively cited by peer reviewed articles. We limited our study to reviews and articles indexed by Scopus and Web of Science to ensure they had been peer reviewed, and to facilitate retrieval. Consulting references in, and citations of, Jones’ 1992 seminal critique of the Hawthorne Effect, we sought additional critiques. Among those available to us, we found only two articles that met all the above criteria. We retrieved the available articles indexed in Scopus and Web of science citing these three critiques. A handful were available as pre-prints. Assessing these citations, we categorized the publications as affirming the effect, as neutral (the category included those not taking a stance on the issue, those that were ambiguous, and those that did not address the effect), and as negative. We separately reviewed the articles before comparing notes. Where our assessments differed we sought an agreement, and if no agreement were reached, we classified the article as neutral. We found that a few authors cited the source correctly as being critical to the Hawthorne Effect, making the reader aware that it is a contentious theory. However, while taking a neutral, or perhaps even negative stance on the issue, they still approached the Hawthorne Effect as if real when discussing their method or results. We categorized these as de facto affirming. We also sought to discern whether the authors of the affirmative citations cited the critical articles as affirming the Hawthorne Effect. We found that the majority of affirmative citations simply referred to a critical article when discussing how to avoid the Hawthorne Effect, thus implicitly presenting it as affirmative. However, a handful of these affirmative citations cited the article as a source for how they understood the Hawthorne Effect. Although not serving an argumentative role, the affirmative context nevertheless left the reader with the impression that the cited source did affirm the effect. We chose to classify also these as instances of citation diversion.

Discussion and conclusion The ratios between articles that took an affirmative stance towards the Hawthorne Effect and those that rejected it, was for Jones roughly 6:1, and for Franke and Kaul 11:1. The reception of Wickström and Bendix appears to be an outlier at 34:1. The citation diversion group was extensive. Out of 197 affirmative citations of Franke and Kaul, 189 cited the critical articles as affirming the Hawthorne Effect. For Jones, the number was 60 of 103, for Wickström and Bendix 155 of 168. Considering the numerous citation diversions, a major explanation for the asymmetry between the affirmative articles and the negative articles appears to be not reading, or not understanding, the cited paper. However, we suspect that additional factors may have contributed to the skewed reception: First, there are incentives for not reiterating the critique if convinced by it. For authors accepting the arguments, the Hawthorne Effect becomes irrelevant, unless, of course, they take issue with the effect specifically. Consequently, we expect them to leave out any mention of the effect, without accounting for their deliberations, nor citing the critical publication, due to common standards of text conciseness and continuity. This can explain the small number of authors reiterating a critical stance towards the theory. Second, the findings may also reflect that the majority of the citers initially were partial to the Hawthorne Effect: Citing while applying the Hawthorne Effect in methodological discussions is presumably more frequent than citing it with the intent of criticizing it. Of course, to assess whether the three articles were successful at communicating their critique of the Hawthorne Effect, we ought to consider the number of readers that has been dissuaded from believing in and using the Hawthorne Effect in their research. For all we know this group is in majority. It is, however, a silent one. When it comes to academic publishing, the affirming articles are dominant on the issue of the Hawthorne Effect, and are likely the major contributors to the forming of the published consensus. These publications, we surmise, will efficiently recruit new believers in the effect, and in turn new affirmative citations in the literature. The findings not only demonstrate that the three efforts at criticizing the Hawthorne Effect to varying degrees were unsuccessful, but they also suggest that if the intention behind the critiques were to reduce the frequency of affirmations of the claim in the scientific corpus, they may have achieved the very opposite

Acknowledgments We owe thanks to Finnur Dellsen, Espen Dragstmo, Anstein Gregersen, Anders Nes, Knut Olav Skarsaune, and Terje Ødegaard for suggestions that has greatly improved the manuscript.