Abstract Objective To investigate the replication validity of biomedical association studies covered by newspapers. Methods We used a database of 4723 primary studies included in 306 meta-analysis articles. These studies associated a risk factor with a disease in three biomedical domains, psychiatry, neurology and four somatic diseases. They were classified into a lifestyle category (e.g. smoking) and a non-lifestyle category (e.g. genetic risk). Using the database Dow Jones Factiva, we investigated the newspaper coverage of each study. Their replication validity was assessed using a comparison with their corresponding meta-analyses. Results Among the 5029 articles of our database, 156 primary studies (of which 63 were lifestyle studies) and 5 meta-analysis articles were reported in 1561 newspaper articles. The percentage of covered studies and the number of newspaper articles per study strongly increased with the impact factor of the journal that published each scientific study. Newspapers almost equally covered initial (5/39 12.8%) and subsequent (58/600 9.7%) lifestyle studies. In contrast, initial non-lifestyle studies were covered more often (48/366 13.1%) than subsequent ones (45/3718 1.2%). Newspapers never covered initial studies reporting null findings and rarely reported subsequent null observations. Only 48.7% of the 156 studies reported by newspapers were confirmed by the corresponding meta-analyses. Initial non-lifestyle studies were less often confirmed (16/48) than subsequent ones (29/45) and than lifestyle studies (31/63). Psychiatric studies covered by newspapers were less often confirmed (10/38) than the neurological (26/41) or somatic (40/77) ones. This is correlated to an even larger coverage of initial studies in psychiatry. Whereas 234 newspaper articles covered the 35 initial studies that were later disconfirmed, only four press articles covered a subsequent null finding and mentioned the refutation of an initial claim. Conclusion Journalists preferentially cover initial findings although they are often contradicted by meta-analyses and rarely inform the public when they are disconfirmed.

Citation: Dumas-Mallet E, Smith A, Boraud T, Gonon F (2017) Poor replication validity of biomedical association studies reported by newspapers. PLoS ONE 12(2): e0172650. https://doi.org/10.1371/journal.pone.0172650 Editor: Jelte M. Wicherts, Tilburg University, NETHERLANDS Received: October 27, 2016; Accepted: February 7, 2017; Published: February 21, 2017 Copyright: © 2017 Dumas-Mallet et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability: All relevant data are within the paper and its Supporting Information files. Funding: The authors received no specific funding for this work. Competing interests: The authors have declared that no competing interests exist.

Introduction Over recent years many scientific editorials have highlighted the "reproducibility crisis" of biomedical findings [1–5]. Empirical studies have actually observed that many initial findings are refuted by subsequent studies, a trend that holds true at all three levels of biomedical research: i) preclinical studies [6, 7], ii) associations between biomarkers or risk factors and diseases [8–11], and iii) clinical trials [12–16]. These empirical studies, as well as theoretical considerations [1], have highlighted that initial findings on a novel scientific question are especially vulnerable to refutation. However, others have argued that it is a mistake to expect 100% replication [17]: “a single study almost never provides definitive resolution for or against an effect and its explanation.” Scientific progress is a cumulative process that reduces uncertainty by combining uncertain initial findings with validation by subsequent studies. In other words, the low replication validity of many initial studies does not hinder the advancement of knowledge as long as science remains self-corrective [18]. However, an unrecognized consequence of the low replication validity of single studies concerns their media coverage. In the case of attention deficit hyperactivity disorder (ADHD), we previously found that newspapers preferentially report initial findings although most are refuted or strongly attenuated by subsequent studies [19]. We examined the replication validity of ten biomedical studies related to ADHD, which were widely reported by the media during the 1990’s. Among them, seven were found to be initial studies and all seven were either refuted or strongly attenuated by subsequent studies [19]. However, the strength of evidence provided by this preliminary study was limited by its sample size. The present study examined the preferential media coverage of early biomedical findings on a larger scale. Another weakness of our preliminary study was related to the heterogeneity of the ten ADHD studies that were examined: one was a pre-clinical study in mice, another estimated ADHD prevalence, three investigated the effectiveness of medication and five investigated the association of ADHD with biomarkers or risk factors. In the present study we did not consider pre-clinical studies because they are rarely included in meta-analyses, thus precluding assessment of their replication validity. We also excluded clinical trials investigating treatment effectiveness because their publication is influenced by stakeholders, in most cases pharmaceutical companies [20, 21], and because their media coverage is biased by this influence [22, 23]. Because this influence represents a parameter that might confound our analysis, we decided to focus on studies associating biomarkers and risk factors with diseases. This type of biomedical research is crucial for improving the understanding, diagnosis and prevention of diseases and is thus often reported by newspapers. We recently built a large database of 595 meta-analyses investigating biomarkers and risk factors associated with four psychiatric disorders, four neurological pathologies and four somatic diseases [11]. Starting from the assumption that each meta-analysis provided the best estimate of the ‘true’ effect size, we showed that most effect sizes reported by initial studies were inflated. In the present study we investigated three questions. First, we quantified and characterized the newspaper coverage of the scientific publications included in our database. Second, we tested whether scientific observations reported by newspapers were confirmed, attenuated or refuted by the corresponding meta-analyses. We explored several factors that might influence the percentage of studies covered by newspapers and the replication validity of scientific observations that are covered: the type of study (initial versus subsequent studies and null findings versus "positive" ones), the prestige of the scientific journal that published them, the biomedical domain and the type of association. In particular, as previous studies showed that newspapers preferentially cover studies associating diseases with lifestyle risk factors (e.g. smoking) [24, 25], we considered this type of association to be a category. Third, we analyzed the possible media coverage of subsequent studies contradicting initial claims that had previously been reported by newspapers.

Methods Scientific studies and their classification The database described in our previous publication [11] included 663 meta-analyses investigating the association between biomarkers, risk factors or cognitive/behavioral observations and twelve pathologies belonging to three biomedical domains: psychiatry (Attention Deficit Hyperactivity Disorder (ADHD), autism, major depression, and schizophrenia), neurology (Alzheimer's and Parkinson's diseases, epilepsy, multiple sclerosis) and four somatic diseases (breast cancer, glaucoma, psoriasis and rheumatoid arthritis). In a preliminary step of the present study we observed that initial studies related to cognitive/behavioral observations were never covered by newspapers. Therefore, the 68 meta-analyses dealing with cognitive/behavioral observations were not considered in the present study. The 595 meta-analyses considered here were reported by 306 meta-analysis articles and included primary observations reported in 4775 scientific articles. As described previously, initial studies were distinguished from subsequent ones on the basis of their publication date (their references are given in our previous study [11]). From this database we excluded primary articles published before 1981. Indeed, we used the Dow Jones Factiva database to test whether these primary articles were covered by newspapers and the Factiva database starts on January 2nd 1981. Thus our final database includes 5029 scientific articles: 405 initial studies, 4218 subsequent studies and 306 meta-analysis articles. In our previous study, these 595 meta-analyses were classified into three types: genetic studies, brain imaging studies and "other" studies. This latter type included a wide variety of biochemical, biophysical and epidemiological studies. Previous studies [24, 25] and a preliminary step of the present study led us to subdivide all association studies into two categories: a lifestyle category corresponding to any association between a pathology and a risk factor on which each subject can individually decide to act (e.g. red meat consumption, smoking, non compulsory vaccination) and a non-lifestyle category. Impact factor of the journals that published scientific studies We used the Journal Citation Reports from The Web of Science to determine the 2012 impact factor of the journals that published the 5029 articles of our database. In our previous study we observed that studies associating ADHD with a biomarker or a risk factor were rarely covered by newspapers if they were published in a scientific journal with an impact factor below five [19]. Therefore, when testing whether each study of our database was echoed by newspapers, we limited our initial search to articles published in scientific journals with an impact factor equal or superior to five. We observed that among the 91 scientific articles reported by newspapers and related to other pathologies than breast cancer, the lowest impact factor was 6.5. We found no press coverage for the 220 articles published in journals with an impact factor between 5 and 6.5. Therefore, it seems unlikely that the 2342 primary studies not related to breast cancer and published in journals with an impact factor below five attracted journalists' attention. As a result, we did not check the media coverage of these 2342 primary studies. Regarding breast cancer, among the 64 scientific articles reported by newspapers, we found 14 articles with an impact factor between 5 and 6.5. Therefore, we extended our search for a possible media coverage of breast cancer studies with an impact factor between 4 and 5. However, we found no media coverage for these 340 articles. Therefore, we did not check the media coverage of the 339 primary studies related to breast cancer with an impact factor below four. In order to test the influence of the impact factor on media coverage, we considered three classes. Prestigious journals with an impact factor ≥ 30 (Nature, Science, JAMA, The Lancet and The New England Journal of Medicine) formed the first class. The limit between the two other classes was adjusted in order to keep the number of covered articles roughly equal in both classes. Thus, the second class included articles published in journals with an impact factor between 10 and 30 and the third one all other articles (IF ≤ 10). Newspaper coverage of scientific studies As previously described [19], we used the Dow Jones Factiva database to find newspaper articles echoing the scientific studies of our database. Briefly, the following keywords were used in each search: "study OR research* OR scientist*". Moreover, each search also used specific keywords that included the name of the pathology, one or several keywords that characterized the study (e.g. "gene", "milk", "birth weight") and was restricted to one month after the publication date of the study. Some searches, particularly about genetic associations, led to many items. In those cases, we added the name of the authors of the study, or of the university to the keywords. We only considered newspaper articles written in English and published in the general press. Articles published in the specialized press (e.g. Pharma Business Week), as well as articles published by any press agency (e.g. Associated Press), were not taken into account. Sometimes newspaper articles found by this search actually highlighted a biomedical finding that was not included in our database of 595 meta-analyses. For example, some newspaper articles underlined risk factors associated with colon cancer, but did not mention the association of the same risk factor with breast cancer although it was also reported in the corresponding scientific article. In these few cases this media coverage was not considered. This search was performed by one author (EDM). Another author (FG) independently checked the possible media coverage of 20% of scientific studies. Both authors always reached agreement on whether any scientific study was echoed or not. A few minor disagreements occurred regarding the number of newspaper articles reporting some studies and they were resolved by discussion. Replication validity of scientific observations reported by newspapers We used our previously published database of 595 meta-analyses to check whether biomedical findings echoed by newspapers were confirmed, or not, by the corresponding meta-analyses. As previously described [11], a primary observation was considered in agreement with the corresponding meta-analysis if it satisfied two criteria: first, both observations were in nominal agreement (i.e. both reported a statistically significant effect in the same direction or the absence of a significant effect) and, second, when both reported a significant effect, the effect size of the primary observation must not be more than twice as large as that of the corresponding meta-analysis (i.e. effect size inflation below 100%). Sometimes single scientific articles reported several associations included in our 595 meta-analyses while newspaper articles that echoed them did not specify which association they highlighted. This is often the case with genetic studies: newspapers reported that new genes have been found to be associated with a disease, whereas the scientific article described significant associations with several single nucleotide polymorphisms. In these cases we considered the replication validity of the most credible association (i.e. with the smallest p value of the statistical test). Thus, we assessed the replication validity of any scientific article covered by newspapers under the most favorable conditions.

Comments We explored the replication validity of association studies echoed by the lay press. Altogether, 48.7% of the studies examined were confirmed by subsequent meta-analyses. However, subpopulation analyses show large variations of this replication rate. Newspapers preferentially echo initial studies over subsequent studies investigating the same question and initial findings covered by newspapers are less often validated by replications (34%) than subsequent findings (56.3%). This observation is consistent with several observational studies reporting that many initial or early biomedical findings are disconfirmed by subsequent studies [6–16]. Thus, the present study further supports our previous conclusion [19]: a major factor that explains the poor replication validity of biomedical studies covered by newspapers is that journalists favor initial findings over replication studies. In agreement with previous studies [24, 25], we observed that newspapers preferentially reported lifestyle association studies linking a pathology to a risk factor on which each reader can act. Non-lifestyle studies related to brain imaging, genetic factor or other inescapable risk factors were less often echoed. This preferential coverage further supports the view that the first journalists' aim is to attract readers' attention. We also observed that newspapers never covered initial findings reporting null findings and under-reported subsequent null findings, particularly regarding non-lifestyle studies. As a consequence, journalists rarely cover null findings that contradict previously echoed studies. Within our database we observed only one interesting exception to this general rule: four newspaper articles covered the meta-analysis by Risch and coworkers and informed the public that it disconfirmed Caspi's initial study, which received much larger coverage (50 articles). All these observations support the conclusion of a case study: "scientific uncertainty per se is not attractive to journalists" (p. 344) [30]. We observed that findings reported by newspapers and related to psychiatric disorders were less often confirmed by meta-analyses than those related to neurological or somatic diseases. We correlated this observation with the fact that the preferential coverage of initial studies is even stronger regarding psychiatry. However, another factor might contribute to the low replication validity of psychiatric studies covered by newspapers: we previously showed that initial association studies linking biomarkers or risk factors to psychiatric disorders were less often confirmed by corresponding meta-analyses than those related to neurological diseases [11]. Likewise, highly cited early studies investigating the effectiveness of psychiatric treatments are often contradicted by subsequent studies [16].

Limitations Although single primary studies have often investigated several associations linking a biomarkers or risk factors to a pathology (e.g. several polymorphisms on the same gene), we restricted our replication analysis to only one association per scientific article. Because we selected the most credible association (i.e. with the highest degree of statistical significance) it is likely that the replication rates we reported here are overestimated. The 4723 primary articles of our database were included in meta-analyses thus enabling us to assess their replication validity. Whether the newspaper coverage of association studies included in meta-analyses is representative of the coverage of all association studies remains an open question. We only considered newspaper articles published within a month following the publication of each scientific study. Indeed, when newspaper articles were published much later, it was often impossible to identify the scientific study they were reporting on with certainty. Consequently, we did not investigate the long-term impact of scientific publications in the lay press, although there is no doubt that certain scientific studies did have such an impact. For example, Caspi's study published in 2003 about a genetic susceptibility to major depression [28] was still cited in the lay press well after its refutation in 2009 by a meta-analysis [29]. However, in relative terms, it is likely that the scientific publications with the strongest long-term impact also received the widest newspaper coverage shortly following their publication. Our data support three conclusions. First, newspapers preferentially cover lifestyle associations over non-lifestyle ones. Second, they strongly favor initial findings over subsequent studies when they cover non-lifestyle associations, but they equally cover initial and subsequent lifestyle studies. Third, among non-lifestyle associations covered by newspapers, initial findings are less often confirmed by meta-analyses than subsequent studies. However, our number of observations is too small to know whether this third conclusion also holds true regarding lifestyle studies. Indeed, our database included only five initial lifestyle studies covered by newspapers. We only studied newspaper articles but ignored television reporting, and this although television is the main source of information on medical sciences for Europeans (European Commission, 2007). Indeed, television programs rarely explicitly report on specific scientific publications [31]. Nevertheless, newspaper articles might indirectly influence the content of television programs by raising the attention of their producers.

Conclusion Biomedical researches mature from early studies that are highly uncertain to scientific consensuses built on subsequent studies and quantified by meta-analyses. Indeed, recent meta-researches have consistently shown that most early studies report inflated effect size or false positive effects [6–16]. Our study shows that many biomedical findings reported by newspapers are disconfirmed by subsequent studies. This is partly due to the fact that newspapers preferentially cover "positive" initial studies rather than subsequent observations, in particular those reporting null findings. Our study also suggests that most journalists from the general press do not know or prefer not to deal with the high degree of uncertainty inherent in early biomedical studies. Importantly, such biased newspaper coverage can have important social consequences [32, 33]. For example, a content analysis of newspaper articles covering Caspi's study showed that they emphasized the genetic side of the gene-by-environment interactions and this discourse deflected public attention away from the considerable impact of social inequalities upon health [34]. Therefore, with Susan Watts [35], we advocate that society "needs science journalism to weigh up the values and the vices of new science" (p. 151). In particular, when preparing a report on a scientific study, journalists should always ask scientists whether it is an initial finding and, if so, they should inform the public that this discovery is still tentative and must be validated by subsequent studies. Larsson and coworkers (2003) have identified the obstacles science journalists meet to accurately cope with uncertainty [36]. In particular, most interviewed journalists feel that it is difficult to find scientists who are independent from authors and who are willing to assist them. Thus, scientists, either as independent experts or as authors, are also responsible for improving the informative value of biomedical reporting in the mass media. In particular, they are responsible for the accuracy of the press releases covering their work and published by scientific editors or universities. This is not only a moral duty but also in the interest of science. Indeed, contrary to common beliefs, reception studies show that scientists are viewed by the public as more trustworthy when news coverage of health research acknowledges its uncertainty, especially if this is attributed to scientific authors [37, 38].

Author Contributions Conceptualization: EDM AS TB FG. Data curation: EDM TB FG. Formal analysis: EDM TB FG. Investigation: EDM FG. Methodology: EDM FG. Project administration: EDM AS TB FG. Supervision: AS TB FG. Validation: FG. Visualization: EDM TB FG. Writing – original draft: EDM FG. Writing – review & editing: EDM AS TB FG.