The problem of bacterial complications post respiratory viral infections

Andrej Egorov1,2 Author Affiliations 1 University of Natural Resources and Life Sciences, Vienna, Austria

2 Mechnikov Research Institute for Vaccines and Sera, Moscow, Russian Federation

# Corresponding author: Andrej Egorov, e-mail: aevirol@gmail.com

Abstract Back to the top

REVIEW

Every person over the course of their lifetime is repeatedly infected by a variety of respiratory viruses that represent risk factors for the development of bacterial complications. The most dangerous among the etiological factors of acute respiratory viral diseases is the influenza A virus. This virus is capable of causing catastrophic pandemics with high mortality mainly due to secondary bacterial pneumonia. As has been shown in numerous recent studies, the main mechanism of provoking bacterial infections irrespective of the type of respiratory virus is the imbalanced response of the antiviral innate immunity – excessive interferon response and uncontrolled inflammation. The probability of severe bacterial complications in the course of acute respiratory viral infections is determined by both the virulence of the virus itself and by the composition of the respiratory microbiota at the time of the viral infection as well as by the genetic characteristics of the organism. The occurrence of severe bacterial complications is also affected by the chronic diseases that have an impact on the regulation of the innate immune response. This review summarizes the current concept of the mechanisms of the development of post viral bacterial complications as well as the potential prevention strategies for these complications.

Review Back to the top

Introduction

According to estimates by the World Health Organization (WHO), acute respiratory viral infections (ARVI) kill 3.9 million people per year [1]. These infections affect all age groups but have the strongest impact on the health of children, the elderly and people with chronic diseases. The risk of ARVI is defined by their ability to provoke viral pneumonia and secondary bacterial infections, e.g. bacterial pneumonia, otitis, meningitis as well as chronic lung diseases. Therefore, the respiratory viruses are involved in about half of all cases of community-acquired pneumonia (CAP) in children. Furthermore, they provoke more than 90% of the cases of bronchiolitis in infants and 85-95% of the cases of the exacerbation of asthma in children. Beyond the pandemic influenza period, ARVIs are the cause of 30-50% of CAP cases, 80% of the exacerbations of asthma, and 20-60% of the exacerbations of chronic obstructive pulmonary disease (COPD) in adults. In general, these infections are one of the top five causes of mortality among the human population in the world, and in many developing countries they are the leading causes of mortality of children under five years of age.

Unlike other respiratory viruses, the influenza A virus poses the greatest threat due to its circulation in different animal species and ability to reassort genomic fragments leading to the rapid modification of the main surface antigens of the virus – hemagglutinin (HA) and neuraminidase (NA) (antigenic shift). This mechanism defines the possibility of the emergence of new antigenic variants of the influenza A viruses, such as H1N1, H2N2, H3N2, H1N1pdm09, which have caused four known pandemics [2, 3].

In the periods between pandemics, influenza viruses also undergo antigenic changes caused by the gradual accumulation of mutations in HA and NA (antigenic drift) that lead to the emergence of seasonal epidemics.

Even now, in the 21st century, when anti-influenza drugs and broad-spectrum antibiotics are available, people with seasonal influenza have a likelihood of developing complications. Thus, both virus-induced acute respiratory distress syndrome (ARDS) and secondary bacterial pneumonia could develop within the first week post viral infection. According to the WHO, influenza causes up to 650,000 deaths worldwide every year [1]. In the US, with a population of about 300 million, the death rate from influenza varies between 16,000 and 56,000 cases per year according to the Centers for Disease Control and Prevention (CDC, Atlanta, GA, USA). In the Russian Federation with a population of about 140 million, mortality from seasonal influenza is estimated at the level of several hundred per year, although there are no official data available in the public domain. The difference in the statistics on the mortality rate from influenza in various countries could be defined by the variation in the calculation methods. For example, the mortality statistics from bacterial pneumonia may not take into account the role of the viral etiological factor as a provoker of a secondary infection. Thus, the individuals who died from bacterial pneumonia may not be included when calculating the mortality rate from influenza even if the bacterial complications were provoked by the primary influenza infection.

Influenza in the absence of antibiotics, antiviral drugs, and vaccines

The pandemic of A(H1N1) influenza virus (Spanish flu) of 1918 was the most fatal influenza epidemic ever recorded. In the course of this pandemic, which happened during the First World War, 50% of the world population was infected resulting in the death of about 40-50 million people [4]. This pandemic mostly affected people who were 20-40 years old and serving in the army. Since the influenza virus was discovered in 1933, at the time of the Spanish pandemic the etiological agent of the disease had not been identified yet. At that time, microbiologists tried to connect the Spanish flu with various bacterial agents, in particular with Haemophilus influenzae (H. influenzae) [5]. Now, due to the availability of the PCR technique, it became possible to identify the genes of the influenza virus in the autopsied and histological specimens of that time and to reconstruct the Spanish H1N1 virus by means of reverse genetics. Surprisingly, the reconstructed virus did not have radical differences in the genetic structure from the viruses that are circulating at present [6]. A subsequent study of the autopsied specimens showed that more than 90% of the analyzed lung tissue samples of those who died during the Spanish pandemic from influenza had signs of bacterial infection [7-9]. It turned out that 95% of the deaths were caused not by the influenza virus itself but by the consecutive bacterial pneumonia. In particular, it was proved that Streptococcus pneumoniae (S. pneumoniae) was the main infectious agent that caused the diagnosed pneumonia [10]. The autopsy showed the presence of S. pneumoniae in the lungs of 44% and in the blood of 33% of the patients who died from the Spanish influenza [10, 11]. Though S. pneumoniae was the leading cause of mortality, other infectious agents like Staphylococcus aureus (S. aureus), H. influenzae and a number of other Streptococcus spp. were also identified in the autopsied lung specimen [5, 7]. Therefore, the catastrophic outcome of the 1918 influenza pandemic could have been defined by the lack of vaccines and antiviral drugs against the influenza virus as well as by the lack of antibiotics that could suppress the secondary bacterial complications.

Pandemic influenza in the era of antibiotics in the absence of vaccination and antiviral drugs

In 1957, the pandemic of the Asian flu affected 40-50% of the people around the globe with estimates of total mortality between 1.5 and 4 million people [13], with the death toll in the United States reaching around 69,800 people [13-15]. The cause of this pandemic was the A(H2N2) strain of the influenza virus [12]. Analysis of the autopsied specimens revealed evidence of bacterial infection in 80% of all the severe and fatal cases [8, 16, 17]. It should be noted that, by this time, such antibiotics as penicillin and streptomycin were already available and used in medical practice in developed countries. However, during this pandemic in the US and a number of other countries, there was a noticeable increase in the number of hospitalizations associated with pneumonia caused mainly by S. pneumoniae, H. influenzae and S. aureus [18]. Similar data were reported in the Netherlands, wherein, of the 148 deaths allegedly caused by the Asian pandemic influenza, 75% were associated with bacterial pneumonia provoked in 15% of cases by S. pneumoniae and in 59% cases by S. aureus [16]. It should be mentioned that the cited bacteriological examination data could be highly distorted since at that time many patients were already treated with antibiotics [19].

The world death toll from the Hong Kong pandemic of 1968-1969 that was caused by the A(H3N2) strain of influenza virus was around 2 million people [20, 21]. A 55% increase in mortality from the respiratory infections with most cases accompanied by secondary bacterial pneumonia was recorded in Great Britain in 1969 [22]. The staphylococcal pneumonia turned out to be the main complication of the H3N2 Hong Kong influenza. For example, from the 129 adults diagnosed with pandemic influenza, pneumonia was found in 16% of cases of which 40% (6% of all 129 cases of influenza) were lethal. S. aureus or Pseudomonas aeruginosa (P. aeruginosa) were found in 75% of the autopsied specimen of the deceased [23]. It should be mentioned that over the course of the Spanish pandemic, most of the lethal cases of the secondary bacterial pneumonia were caused by S. pneumonia, but during the pandemics of 1957 and 1968, S. aureus became the prevailing etiological agent of the secondary bacterial pneumonia. This could be explained by the increased antibiotic resistance of S. aureus due to the widespread use of antibiotics by the population at that time.

Pandemic influenza in the presence of influenza vaccines, antiviral drugs, and antibiotics

The outbreak of influenza A(H1N1)pdm09 virus in 2009 spread to 41 countries in the course of four weeks and became a pandemic [20, 24]. The H1N1 component of the seasonal influenza vaccines produced for 2009 did not antigenically match the newly emerged pandemic strain A(H1N1)pdm09. It is believed that the total death toll from this pandemic was around 284,000 people worldwide [25]. The common causes of death in the course of this pandemic in the US were the secondary bacterial infections induced mostly by S. pneumoniae and S. pyogenes [26, 27]. Other US studies have shown that in almost 30% of the 77 lethal cases the influenza infection was accompanied by bacterial infections 46% of which were caused by S. pneumoniae, 9% by S. aureus, and 1% by H. influenzae. Palacios et al. examined the nasopharyngeal swabs of almost 200 patients with pandemic influenza and found H. influenzae in 52% of the samples, S. pneumoniae in 31%, and S. aureus in 18% of samples, respectively [28]. The severity of the disease was most often associated with the secondary infection induced by S. pneumoniae.

In another study of 838 critically ill children in the US, it was shown that in 72 hours following the hospitalization to the intensive care unit, bacterial superinfection developed in 33% of the patients, 48% of which was caused by methicillin-resistant S. aureus (MRSA), 5.5% of the cases by S. pneumoniae, and 5% by H. influenzae [29].

Therefore, despite the widespread introduction of seasonal influenza vaccines, antiviral drugs, and antibiotics, the problem of bacterial complications in the course of influenza has not ceased to be relevant. Moreover, due to the development of the bacterial flora resistance to modern antibiotics this problem could be exacerbated in the case of a new influenza pandemic [30].

Respiratory viral infections of non-influenza etiology as provokers of the secondary bacterial pneumonia

State-of-the-art diagnostics allows scientists to analyze the bacterial complications in various non-influenza respiratory infections. Table 1 summarizes numerous studies that show the relationship of respiratory viral pathogens with certain bacteria that can cause pneumonia, otitis media, sinusitis, and meningitis as complications. Despite the fact that other respiratory viruses are not able to cause pandemics, these viruses proved to be as dangerous as influenza in provoking secondary bacterial complications. For example, the representative of the family Paramyxoviridae, human metapneumovirus (hMPV), can cause severe complications as influenza virus including both the viral pneumonia with the development of ARDS and bacterial pneumonia [31]. Another representative of this family – the respiratory syncytial virus (RSV) – turned out to be even more dangerous than influenza in terms of the mortality rate among the hospitalized patients diagnosed with pneumonia. Thus, the mortality from all of the causes in 20 days after hospitalization was higher in patients with an RSV infection (18.4%) than in patients with influenza (6.7%) [32]. Unfortunately, there are no vaccines and effective antiviral drugs for the prevention and treatment of pneumoviruses as well as for the most respiratory viruses of non-influenza etiology.

Table 1. Respiratory viruses and the bacterial infections associated with them

Virus Pathogens associated with secondary infection Reference Influenza virus Streptococcus pneumoniae [7], [33-40] Staphylococcus aureus Staphylococcus pyogenes Haemophilus influenzae Moraxella catarrhalis Neisseria meningitidis RSV Streptococcus pneumoniae [41] Adenovirus Streptococcus pneumoniae [42] Haemophilus influenzae Moraxella catarrhalis Human Rhinovirus Streptococcus pneumoniae [43-45] Staphylococcus aureus Haemophilus influenzae Moraxella catarrhalis Parainfluenza virus Moraxella catarrhalis [46] Streptococcus pneumoniae hMPV Streptococcus pneumoniae [47]

While the observational studies conducted in different regions of the world on the relationship of viral and bacterial infections lead to consistent conclusions, they can differ in the evaluation of the roles various pathogens play in the development of bacterial CAP. This can be explained by the significant influence of the climate conditions and genetic characteristics of the population on the prevalence of particular viral and bacterial pathogens.

In the US, 1,024 patients with CAP were examined and compared with a group of 759 control patients without symptoms [48]. It turned out that influenza viruses, RSVs, hMPVs, and to a lesser extent parainfluenza viruses and coronaviruses were significantly more often isolated from the group of patients diagnosed with pneumonia, regardless of the patient’s age. At the same time, rhinoviruses were a common cause of CAP in adults but were practically not diagnosed in children with pneumonia. In contrast, adenoviruses were associated with CAP only in children younger than 2 years old.

In a study performed in Japan that involved 2,617 elderly (≥65 years old) patients, diagnosed with chronic respiratory or other concomitant diseases, the viral pathogens were isolated in 23.1% of cases [49]. Rhinoviruses were the most common cause of pneumonia (9.8%), while both influenza and RSV were found in 3.9% of cases. RSV was most often diagnosed in individuals with chronic respiratory diseases, while other viruses were associated with pneumonia, regardless of the presence of respiratory and other chronic diseases. In general, viral infections did not predetermine an increased risk of mortality in patients with pneumonia, but influenza led to a threefold increase in mortality in patients with chronic respiratory diseases. Therefore, the contribution of the viral infection to mortality from pneumonia depends on the type of viral agent and the presence of concomitant diseases.

The South Korean meta-analytical study conducted on 5,298 patients of different age groups showed that the provocation of bacterial superinfection is typical for all of the viral respiratory infections [50]. However, the type of bacterial pathogen that causes the secondary infection could depend on both the type of the viral infection and the age of the patient. For example, influenza A and B virus infections, rhinovirus, and hMPV caused pneumonia in adult patients that was associated with S. aureus, whereas coronavirus, parainfluenza virus, and RSV resulted in pneumonia associated with Gram-negative bacteria Klebsiella spp. and Acinetobacter spp. Mycoplasma infection (M. pneumonia) was the most common cause of bacterial pneumonia in children regardless of the type of previous virus infection. These studies stress the importance of viral diagnostics in patients with secondary bacterial pneumonia in terms of choosing the right antibiotic therapy. For example, the post-viral pneumonia in infants and children should be treated with antibiotics active against M. pneumoniae (macrolides). Adults with influenza, hMPV, or rhinovirus infection should be treated with antibiotics that efficiently kill staphylococcal bacteria, whereas antibiotics that are active against a broad range of Gram-negative bacteria should be prescribed in cases of infection with coronavirus and parainfluenza virus.

However, as it was noted hereinabove, the relationship between the population genotype and climate conditions on the one side and the spectrum of the colonizing microbial flora on the other side should be studied further in order to develop adequate medical standards for the treatment of respiratory infections in patients across different countries.

Pathogenesis of secondary bacterial infections following ARVI

The human body is a carrier of a variety of bacterial species, collectively referred to as a microbiota [51]. Normally these bacteria, which are called commensal strains, live in a mutually beneficial symbiosis with the host and provide a variety of useful functions for the host, such as protecting the host organism by means of competition with pathogenic bacteria. Usually the small quantity of pathogenic bacteria, including S. pneumoniae, S. aureus, H. influenzae, S. pyogenes, Moraxella catarrhalis, and MRSA, are present in the microbiota of the human upper respiratory tract without serious consequences for the organism [52]. The shift in the microbiota content toward the increase of pathogenic strains can occur in the course of the viral respiratory infections due to the suppression of antibacterial protection factors. As a result, the secondary bacterial infections of the respiratory tract, provoked by the viral infection, are most often caused by resident pathogenic bacteria rather than by the external infection [53, 54].

Many factors that contribute to bacterial colonization in the course of the viral infections are known to date:

The damage of the integrity of the mucous membranes by the viral and bacterial enzymes, for example, by the influenza virus NA [55]. The increase of bacterial adhesion to the cells that are infected by virus [54, 56, 57]. Disruption of the ciliary epithelium over the course of the viral infection, leading to the reduced self-purification ability of the respiratory tract mucous membranes [58, 59]. Induction of IFNs (I, II, and III types) and cytokines in the course of ARVI that reduce the effectiveness of antibacterial immunity [60-64]. Impairment of antibacterial activity of neutrophils and macrophages due to the reduction of their quantity and decreased phagocytic activity in the nidus of a viral infection [65-67].

The effect of most of these factors depends not only on the nature of the viral pathogen, but also on the type of the colonizing bacteria. For example, it is known that S. aureus can enhance the viral reproduction as a result of suppression of the type I IFN signal transmission by inhibiting the STAT-1–STAT-2 dimerization [68]. In the course of RSV infection in children, the severity of the disease could be defined by the prevalence of S. pneumoniae and H. influenzae in the microbiota before the start of the viral disease [69].

Nevertheless, the results of animal experiments and clinical observations show that the colonization of the respiratory tract by the pathogenic bacterial strains is most often triggered by viral infections. Primary infection of mice with a sublethal dose of bacteria followed by infection with the sublethal dose of influenza virus did not lead to secondary bacterial pneumonia, while the reverse sequence of infections caused pneumonia with a lethal outcome [70-72]. In the case of the viral infection, a bacterial complication occurred most often on the 4th day for animals post viral infection or on the 7th day for the humans post clinical manifestation of the viral infection [42, 73]. In this case, regardless of the type of viral pathogen, the type I IFN response plays a major role, focusing the action of the innate immunity system on the fight against viral infection, but at the same time reducing the activity of antibacterial defense factors such as phagocytosis or the production of antibacterial peptides.

The role of IFN response in the pathogenesis of the secondary bacterial infections

Despite the differences in the structural and genetic organization of the viruses of different families, they share a common property – all of them are intracellular pathogens that induce the type I IFN response. The innate immunity system recognizes a viral infection by means of the Toll-like receptors located in the cell membrane or cytoplasmic RIG-1 and MDA5 pattern recognition receptors (PRRs). These receptors detect the components of the pathogens, such as double stranded RNA (dsRNA) or liposaccharides (LPS) that are atypical for the normal cells but are present in the nidus of the infection. Recognition of a viral infection leads to the induction of a cytokine response, first of all of IFNs of I, II, and III types. The system of innate antiviral immunity is described in detail in reviews [74, 75]. The protective effects of IFNs are due to their ability to induce the synthesis of antiviral proteins in the infected and surrounding cells as well as due to regulation of activation and migration of the innate immunity cells that define the development of the specific B- and T-cell immune response [76-78].

On the other hand, a large body of the recent experimental data shows that the effect of the IFN response in the course of ARVI could be detrimental. In the case of the progressive development of infection, the continuous and excessive IFN production can lead to the development of inflammation due to the hyperproduction of chemokines, such as CCL2 and CXCL10, and the infiltration of alveoli with pro-inflammatory monocytes/macrophages and plasmacytoid dendritic cells. In this case, the entry of viral and bacterial ligands of Toll-like receptors like LPS or dsRNA into the inflammation zone could result in the enhanced production of proinflammatory cytokines by these cells with the possibility of development of so-called cytokine storm [79]. Then, the uncontrolled production of pro-inflammatory cytokines and excessive infiltration of lung tissues by cells of innate immunity could become the leading cause of lung damage [80, 81].

The expression of the proapoptotic ligand TRIAL, induced by type I IFN, also results in the massive apoptosis of epithelial cells that contribute to the development of ARDS [82].

An important risk factor in the development of the uncontrolled inflammation is the genetic predisposition of the organism to the regulation of the IFN response. For example, DBA mice, which produce high levels of type I IFN in response to an influenza infection, show a high mortality rate when infected with doses that are safe for C57BL/6 mice with moderate INF production [83, 84]. People with Down’s syndrome have an increased sensitivity to respiratory infections and a high likelihood of serious complications associated with genetically determined hyperproduction of IFN α/γ in response to a viral infection [85].

In addition to cytokine storm stimulation, type I IFNs could be involved in the pathogenesis of the secondary bacterial infection [86, 87]. The development of the IFN response over the course of a viral infection leads to decreases in both the migration and phagocytic function of neutrophils and macrophages by proapoptotic mechanisms or by means of epigenetic regulation of their functions [61, 88, 89].

Type I IFNs are also capable of suppressing the antibacterial defense by inhibiting the production of IL-17 by T-cells [63, 90]. The lack of an IL-17 response leads to the reduction in the production of antibacterial peptides such as lipocalin 2 and BPIFA1 that in turn results in an increase in bacterial growth in the nidus of viral infection [87].

The fact that genetically modified mice deficient in type I IFN receptors (IFNAR) were more resistant to the development of ARDS, bacterial pneumonia, or sepsis compared to mice with normal IFN signaling proves the negative effect of IFN overexpression [61, 62, 84, 91, 92]. The use of a poly (I:C) IFN inducer prior to the infection of mice with pneumococcus resulted in a 100-fold increase in the colonization of the lungs by the injected bacteria [64].

Therefore, the development of severe complications post ARVI could depend not only on the virulence of the virus itself, or on the composition of the respiratory microbiota at the time of the viral infection, but also on the ability of the organism to regulate the intensity of the innate immunity reactions, especially the IFN response.

The role of antiviral vaccination and antiviral therapy in the prevention of the secondary bacterial complications

At present, there are numerous pieces of evidence that antiviral vaccination may reduce the risk of bacterial complications post ARVI. It was shown that immunization with an inactivated influenza vaccine significantly reduces the severity of bacterial superinfection in mice infected with influenza [93]. In another study, it was shown that the immunization of mice with inactivated or live influenza vaccines could equally reduce the lethality of the viral and subsequent bacterial infection, but at the same time it could not limit the growth of S. pyogenes in the lungs. In this case, it was shown that influenza vaccination resulting in the reduced viral load, leads to the reduction of pro-inflammatory cytokines and IFNγ induction in the lungs during the bacterial superinfection [94]. Unfortunately, there is little information on the effect of influenza vaccination in humans on secondary bacterial complications. This is explained by the complexity of such studies since it would require a parallel assessment of antiviral and antibacterial immunity to a large number of potential pathogens. However, in a limited study conducted in the United States among the military personnel, the influenza vaccine reduced the rate of morbidity caused by S. pyogenes [95]. Similarly, the vaccination of children with live cold-adapted vaccine showed a decrease in the incidence of otitis media [96].

Thus, the vaccination against respiratory viruses can help prevent secondary bacterial complications due to the various mechanisms of nonspecific protection by the modulation of the innate immunity. It is reasonable to assume that the decreasing of the viral reproduction due to effective vaccination can lead to the reduction of antiviral IFN response and consequently to limiting the negative IFN action on the effectors of antibacterial defense. In addition, a decrease of viral concentration can help to preserve the respiratory epithelium and decrease the adhesion of bacterial cells [56].

The effect of antiviral drugs, such as NA inhibitors, on prevention of the secondary bacterial infections could also be related to diminished IFN response due to inhibition of virus replication. Studies on animal models showed that NA inhibitors could reduce the susceptibility of animals infected with influenza to the secondary bacterial pneumonia [97]. Clinical trials in adults and children showed that the administration of NA inhibitors (zanamivir and oseltamivir) led to the reduction in the frequency of bronchitis and consequently to fewer antibiotic prescriptions [98-101].

Conclusion

The discovery of antibiotics and the development of influenza and bacterial vaccines have significantly reduced the chances of recurrence of catastrophic pandemics like the Spanish influenza of 1918 [102]. However, influenza and other respiratory viruses remain among the main causes of bacterial complications and of CAP. On the other hand, the increasing resistance of bacterial pathogens to modern antibiotics due to their widespread use is becoming increasingly more alarming.

Unfortunately, at present, there are practically no vaccines or antiviral drugs for the prevention and treatment of respiratory viruses available except for influenza. Despite the progress made in the development of pneumococcal and other antibacterial vaccines and extensive effort to develop vaccines against respiratory viruses, it is unlikely that an effective complex vaccination system will be available against common viral and bacterial respiratory pathogens in the foreseeable future. The creation of specific antiviral drugs against different groups of respiratory viruses that are effective on the late stage of infection has a low probability of success. Therefore, there is an urgent need for the development of pathogenetic drugs reducing the reactions of innate immunity that are weakening the antibacterial protection factors in the course of the viral infection. One of the possible approaches may be to search for drugs that prevent the excessive production of IFN at the peak of a viral infection. The dampening of excess IFN signaling can become an important therapeutic approach to preventing both cytokine storm effects and the development of secondary bacterial complications in the areas adjacent to the respiratory tract [79, 82, 103-105].

Acknowledgements Back to the top

This work was supported by the RSF-FWF grant No 18-45-05002 (A VLP approach to combat post influenza bacterial infections).

Conflict of interest Back to the top

The author does not pursue commercial or financial interests.

Citation Back to the top

A. Egorov. The Problem of Bacterial Complications Post Respiratory Viral Infections. MIR J, 2018; 5(1), 12-21, doi: 10.18527/2500-2236-2018-5-1-12-21.

Copyright Back to the top

© 2018 Egorov. This is an open access article distributed under the terms of the Creative Commons AttributionNonCommercial-ShareAlike 4.0 International Public License (CC-BY-NC-SA), which permits unrestricted use, distribution, and reproduction in any medium, as long as the material is not used for commercial purposes, provided the original author and source are cited.