In contrast, at present, open access (OA) journals are universally available. Libraries have no need to subscribe, and researchers can access OA articles freely. Expectedly, OA publication is associated with increased citations 14 – 19 , so researchers are likely to prefer this path. What is not known is whether, within OA journals, increasing IF is associated with increasing citations, as it is for non-OA journals. Yet this information is important, since cost of publishing in an open access journal is high and increases with the journal’s IF. Should a researcher, or a sponsor, pay good money for publication in a higher IF OA journal if the IF will not influence citations?

In the past, most libraries could possess only a limited number of journals, and librarians used the IF to decide which journals to buy 3 , 5 – 7 . Consequently, high IF journals were more likely to be purchased, read, and cited. With low IF journals, availability was a constraint. Scientists, wanting a greater audience for their research, preferred to publish in high IF journals. There was plenty of evidence that publishing in a higher IF journal resulted in more citations 8 – 13 .

A journal’s impact factor (IF) has long been used as a measure of the quality of a journal 1 . Today, the IF is used as a tool to assess researchers for employment, career promotion, and funding 2 – 4 .

IBM ® SPSS ® Statistics (version 22.0) software was used to conduct the statistical data analyses on the dataset (Dataset 1, doi: 10.7910/DVN/XR6MR9 22 ). OA journals were compared to non-OA journals for overall IF and citations over 5 years. Normality for each independent variable and dependent variable was assessed using the “Kologorov-Smirnov” test, which showed that citations were not normally distributed (p< 0.05). Consequently, non-parametric univariate analysis was carried out using the “Mann-Whitney” test. Linear regression was performed before and after logarithmic transformation of the data.

The articles were scanned for citations as listed in Web of Science, Scopus, and Google Scholar databases. The search period was extended up to 2016, allowing for five years of publication time, with the assumption that citations over five years provide a better estimate of the impact of a paper than citations over two years 5 . Only journal citations were included in the counts; citations in books, theses, and government documents were excluded to conform with the Web of Science policy 21 . We exported citation data from the three databases into .csv files, and imported these into a Microsoft Excel sheet. Duplicates were excluded. Citations that appeared in two language versions of the same paper were counted as one.

In other words, the non-OA article could, in theory, only be read by someone with a subscription. Within non-OA journals, we excluded articles if their journals allowed free access to all articles any time after publication. We further excluded articles published in hybrid non-OA journals if over 20% of their articles were freely available (for this, we counted 100 successive 2011 articles in that journal, and ensured that fewer than 20 were marked as freely accessible). In other words, we attempted to ensure that the non-OA journal was true non-OA, and its IF would properly represent the IF of a non-OA journal ( Figure 1 ). Finally, we also excluded articles if their journal did not have a measurable Web of Science IF for 2011.

In order to have a 5-year follow up for citations, we chose 2011 as the publication year of articles included in this study, and restricted our source articles to those published in January 2011. We found 3,742 RCTs, and saved them into a Microsoft Excel file. The IF of their journals were derived from the Thompson Reuters’ Web of Science database.

We first conducted a pilot study to estimate required sample size. For this purpose, 57 randomized controlled trials (RCTs) were extracted from PubMed, and scanned for citations as listed in the Web of Science. PubMed was chosen to look for source articles because most researchers start their search on PubMed 20 . Within this pilot group, for OA articles the mean citations were 12.0±8.81; for non-OA articles the mean citations were 7.14±6.89. The estimated sample size, at α= 0.05 and β=0.2, was 58 articles per group, which we rounded up to an intended 100 articles per group.

In view of the skew, we repeated the regression analysis after log 10 transformation of the citation data. The data became normally distributed after transformation. The outcome was roughly similar to the pre-transformation results ( Table 3 ).

We calculated the linear regression coefficient between IF and citations. The crude β regression coefficient was 0.297 ( Table 2 ). We then calculated the regression values separately for OA and non-OA publications. There was very little correlation between IF and citations for OA publications. The five-year citations increased by 3.3 for every unit increase in IF. There was, however, significant correlation between citations and IF in non-OA publications, which showed a rise in five-year citations by 4.3 for every unit increase in IF ( Table 2 ).

Discussion

The IF served an important function in the pre-internet era. Libraries needed to decide which journals to buy. With limited budgets, especially in poorer countries, they purchased only a few of the highest IF journals7,23–25. In a self-propagating mechanism, the higher IF journals continued to be better read and cited, and were purchased more often. To quote Peter Suber24, “Prestige even feeds prestige. Journal prestige attracts readers, and helps justify library decisions to spend part of their limited budget on a subscription. The growth in readers and subscribers directly boosts prestige.”

With time, the IF became widely used as a measure of the quality of a journal, author, and paper21,24. Universities rewarded faculty who published in high-IF journals. Promotion and tenure committees, as well as funding agencies, preferred authors who had published papers in high-IF journals24. Researchers thus were driven to publish their best papers in high-IF journals. Instead of the content identifying the journal, the journal began to identify the content.

Today, the game has changed and the efficiency of the internet has lead to the proliferation of OA journals. Libraries do not need to make any choices at all; the reader just needs to decide which paper is relevant and read it. This has diminished at least one purpose served by the IF: to help institutions decide which journals to buy. It also raises two questions. The first is: Are publications in OA journals more likely to be cited than those in non-OA journals? The second is: Will a higher IF lead to more citations?

Citations in OA and non-OA journals OA journals are always available to all—this is their advantage over non-OA journals. Consequently, one would expect that an article published in an OA journal would be more easily accessible, more widely read, and therefore more often cited. Research has proved that this is indeed true14,18,26. Our data has also shown that articles published in OA journals are associated with more citations than those published in non-OA journals—by a factor of 1.3. Although statistically significant, this increase in citations was slightly lower than that shown by others. Antelman14, found that open access publications in various specialties (philosophy, political science, engineering, mathematics) were associated with increased citation rates by a factor of 1.45–1.9. Freely accessible articles had 1.5 times higher citation rates than non-OA articles. Kousha and Abdoli18 showed that citation rates of OA publications were higher by a factor of 1.9, giving them a clear advantage. However, these other authors compared OA articles and non-OA articles, rather than OA journals and non-OA journals. Our data is different as it compares the number of citations of publications in OA journals with citations of publications in non-OA journals. This leads us on to the next question: Is the expectation of more citations with a higher IF being fulfilled?

Correlation between citations and IF At the start of the study we had expected to see a significant correlation between IF and the number of future citations, believing that increasing IF indicated improved quality of journal and article. For OA journals the correlation, however, was poor and insignificant (r s =0.187, p=0.060). We believe that it is safe to say an OA journal’s IF contributes little to an article’s future citations. In contrast, the relationship between citations and IF was strong for non-OA publications. Our correlation coefficient for non-OA publications (0.514), closely matched the values reported by Judge et al (0.44)12, Piwowar and Vision (0.45)27, Vanclay (0.56)11, and Leimu and Koricheva (0.62)28. Thus, despite using different databases, particularly Google Scholar, the citation rate in our study showed a moderate (yet statistically significant) correlation with the IF in our study. This validates our methods, and strengthens the findings about OA publications.

Improvement in citations with increasing IF Linear regression analysis indicated a very real relationship between citations and IF for non-OA publications. The expected citations rise at an approximate rate of one citation per year per rise in impact factor—a change that is consistent with the very definition of the impact factor. This result was quite similar to the findings reported by Vanclay11 and by Perneger29. In contrast, publishing in an OA journal with a higher IF did not result in significantly increased citations. For every 1 unit rise in IF, the data showed a rise of just over 3 citations in five years; using the log 10 transformed data the rise was even lower at low IFs. We could not compare our results to those of other authors, as we were unable to find a publication that correlated IF with citations exclusively for OA journals. We are unable to comment on whether any other variable is a better predictor for an article’s citations compared to the IF, since we did not analyze other factors. Nevertheless, it is reasonable to presume that the article’s quality and relevance will influence the citations much more than IF will. Even for non-OA publications, the citations of an article are likely to be strongly influenced by other factors including the quality of the article, and not by the IF alone. This, of course, is well established4,11,30.

Publishing in OA and non-OA journals Since OA publications are cited more often, it seems logical that a researcher should publish in an OA journal. Should an author search for a high-IF OA journal? An author may reasonably expect about 14 citations in five years, regardless of the IF, and these would rise to about 20 if the OA journal’s IF was 2 (from 11 to 15 if we use the log 10 transformed data). With a rise in IF from 0 to 4, the total citations would not even double. And unlike non-OA journals, OA journals charge the author, and, in general, the higher the journal’s IF, the higher the cost. BioMed Central journals with IFs higher than 2 typically charge article-processing fees of about 2000 euros. Even if the journal’s IF contributes to a higher readership and citation rate — which is questionable, considering the low r2 value — it is doubtful whether the few extra citations are worth the cost. In contrast to OA journals, the number of citations for an article published in a non-OA journal with IF of 4 will be thrice as many as those published in a non-OA journal with an IF of 0. So it makes sense to select as high an IF as possible when publishing an article in a non-OA journal, particularly since non-OA journals charge their readers, and not their contributors.