The use of citation counts in research evaluation is based on the norm that “not citing relevant prior work constituted a violation” (Zuckerman 2018). However, the focus on citation counts in quantitative research evaluation involves the danger of forgetting that a citation does not stand alone. It is embedded in the citing text which leads to different meanings of citations. The determination of different meanings is the objectives of citation context analyses. One goal of these studies is to develop methods allowing more detailed and informative citation analyses: which are, e.g., the papers in a specific field introducing important concepts which are frequently, intensively, and controversially discussed in a field? Quantitative (e.g., by using natural language processing techniques) and qualitative (e.g., classic close reading method) approaches can be performed for citation context analyses (Petrovich 2018).

In recent years, the full texts of papers are increasingly available electronically which opens up the possibility of quantitatively investigating citation contexts in more detail. According to Lamers et al. (2018) “of particular interest … is to determine if systematic analysis of citation context can help shed light on what role previous literature plays when cited in new publications, how new authors use past literature to further their own arguments, and whether we can disentangle disciplinary modes of knowledge accumulation from more general archetypes of contributions made to the scientific landscape” (p. 1115). Following the notion of a multi-dimensional perspective on citation impact proposed by Bu et al. (2019), we introduced in this study the CCA method for determining the citation impact that certain published concepts have had on the scientific community. As Petrovich (2018) notes: “citation context analysis seems particularly suitable to clarify the fine-grained structure of the knowledge accumulation process” (p. 1127). CCA can be used to determine the importance of scientific concepts published in highly-cited landmark publications.

It is worth reflecting on which of the concepts in Kuhn’s and Popper’s landmark books have left the largest impact, and on which fields of study. It is not surprising that Kuhn’s “paradigm” concept has had a significant impact. What is surprising is that it may have had such a disproportionately large impact compared to Kuhn’s other concepts. The paradigm concept accounts for around 40% of the citations to Kuhn’s work. And its impact seems resilient across all FOSs and over time. It is also not surprising that the second and third most popular concepts derived from Kuhn’s book are “normal science” and “scientific revolution”. Kuhn argued that both periods are an integral part of the development of a scientific field. Even though his concern was with the natural sciences, many scholars in the social sciences looked at their own fields of study in an effort to determine the extent to which Kuhn’s model of scientific development describes their own field (see Wray 2017).

Further investigation is needed in order to explain our finding that the concept of “normal science” may have had a slightly greater impact in mathematics (6.85%) than in physics (4.39%). One conjecture is that conceptual innovations in mathematics augment the existing knowledge, rather than replace what was regarded as secure knowledge before. For example, the discovery or development of non-Euclidean geometries did not render Euclidean geometry obsolete. In this way, all innovations in mathematics are continuous with a single normal scientific research tradition. But developments in physics are not like this. Newton’s physical theory led to the rejection of the contact physics associated with Galileo and Descartes. About two centuries later, Albert Einstein’s theory of general relativity led to a generalization of Newton’s gravitational theory. But Einstein’s innovations significantly changed our understanding of central concepts, such as “mass”, which is now regarded as convertible to energy, a significant change from the classical understanding of the concept. And the photon theory of light had a similar effect on the theories of light accepted by earlier generations of physicists. Normal scientific research traditions are interrupted in physics, as new theories replace older theories. It is worth remembering that Kuhn’s cyclical theory of scientific change was explicitly designed with the natural sciences in mind. He did not purport to be describing the dynamics of conceptual change in either the formal sciences, like mathematics, or the social sciences.

Similarly, with respect to Popper, it is not surprising that “falsification” seems to be the most used concept derived from his books. Falsification, after all, is the cornerstone of Popper’s critical rationalism. According to Popper (1934, 1959, 1962), the only way scientists can advance our scientific knowledge is by (i) attempting to falsify hypotheses, and (ii) rejecting those that are falsified. According to Popper (1934, 1959, 1962), falsifiability is also what distinguishes a science from a pseudo-science. Further investigation would be needed in order to explain why “corroboration” is used more in biology than in other disciplines.

Our finding points to a greater comparative impact for Kuhn’s book than Popper’s books in sociology, which is not surprising for two reasons. First, Kuhn (1962) develops a theory of science and scientific change that gives special attention to the changing social dynamics in scientific fields (see Wray 2011). The breakdown of the consensus in a scientific field that characterizes a pending scientific revolution or paradigm change is explicitly described in social terms. The field is described as being in crisis as the consensus breaks down. Even in Kuhn’s discussion of scientific revolutions, he explicitly draws a comparison between scientific revolutions and political revolutions. It is not surprising that this sort of analysis would appeal to sociologists. Popper, on the other hand, focuses narrowly on analyzing the logic of science, in keeping with the positivist tradition in which he both grew up and reacted against. Second, Popper had expressed disdain for the social sciences (see Wray 2017). Kuhn, on the other hand, avoided saying much about the social sciences.

Our finding that Popper’s books seem to have a greater impact than Kuhn’s book in mathematics is also not surprising. Kuhn, after all, was only concerned with the empirical natural sciences, not formal sciences, like mathematics. And Popper explicitly discusses probability theory in his book, thus making it relevant to at least some mathematicians, specifically those working in probability theory.

The fact that the level of uncertainty associated with the key concepts in Kuhn’s book decreases as we approach the present is not especially surprising. It could easily be a function of the fact that people now seem to have settled on how the terms should be interpreted or applied. Similar remarks explain the trends with respect to the uncertainty associated with the use of the key terms in Popper’s books. These books have existed long enough to have achieved canonical status in philosophy of science, and standardizing interpretations.

We have some concern about our method of measuring uncertainty and the level of uncertainty associated with two of Kuhn’s concepts. The term “crisis”, as Kuhn uses the term, is meant to convey a period of uncertainty in a scientific field, as anomalies persistently resist normalization. So the words we use to measure uncertainty may not be catching the uncertainty in the use of the term crisis, but rather the uncertainty associated with the period of crisis in a field. And a similar concern arises with respect to “normal science”. In periods of normal science, scientists take the conceptual scheme or theory for granted. The working assumption is that the theory reflects the structure of the world. Thus, some of the words we use to measure uncertainty may not be catching the uncertainty of the use of the term normal science, but rather the uncertainty that is associated with periods of normal science. The authors citing Kuhn may be using such words to describe normal science in their citances, rather than expressing a relatively low degree of uncertainty with Kuhn’s view of normal science. A more detailed study of these citances would need to be conducted to determine whether the uncertainty we measure reflects the relative uncertainty of the authors using the concepts.

As noted above, we recognize that our study has some limitations. The most severe limitation is probably the representativeness of the sample (see “Statistics” section): the MA database does not contain citation context information for all citing papers. For Popper, 14.4% of the citing papers have citation context information available. In the case of Kuhn, 12.2% of the citing papers have citation context information available. A less severe limitation is the selection of concepts and search terms for the concepts and the uncertainty detected. Here, also the wording and length of the citation context determines our ability to properly assign citation contexts to concepts and determine their uncertainty. Finally, a less worrying limitation is the fact that some publications are not assigned to FOSs in MA. As 99.7% of papers citing Popper’s books and 99.5% of papers citing Kuhn’s book were assigned to a level 0 FOS, this limitation seems negligible.

FOSs in MA are assigned algorithmically on the paper-basis. The quality and details about the algorithm of the FOS assignment are unclear (the same holds true for MA’s indexing criteria). Algorithmic FOS assignments may or may not be accurate. The accuracy of an algorithm based on direct citation relations has been questioned (Haunschild et al. 2018a, b). But a recent case study on computer science publications reported promising results regarding the MA FOSs (Scheidsteger et al. 2018). However, a large-scale comparison has yet to be conducted. Despite these limitations, we are confident that CCA is a very interesting and useful method. The books by Kuhn and Popper were used to show how the CCA method can be applied. If the missing citation context information is distributed randomly (without bias towards concepts, uncertainty words, publication years, and FOSs) our results should remain valid despite the mentioned limitations.