Key Points

Question What proportion of recommendations in current American College of Cardiology/American Heart Association (ACC/AHA) and European Society of Cardiology (ESC) guidelines are supported by evidence from multiple randomized controlled trials (RCTs), and how has this changed over the past 10 years?

Findings In this systematic review of 51 current guideline documents that included 6329 recommendations, 8.5% of recommendations in ACC/AHA guidelines and 14.3% of recommendations in ESC guidelines were classified as level of evidence A (supported by evidence from multiple RCTs), compared with 11.5% of recommendations in a systematic review of ACC/AHA guidelines conducted in 2009.

Meaning Among recommendations in major cardiovascular society guidelines from 2008 to 2018, the proportion supported by evidence from RCTs remains small.

Abstract

Importance Clinical decisions are ideally based on evidence generated from multiple randomized controlled trials (RCTs) evaluating clinical outcomes, but historically, few clinical guideline recommendations have been based entirely on this type of evidence.

Objective To determine the class and level of evidence (LOE) supporting current major cardiovascular society guideline recommendations, and changes in LOE over time.

Data Sources Current American College of Cardiology/American Heart Association (ACC/AHA) and European Society of Cardiology (ESC) clinical guideline documents (2008-2018), as identified on cardiovascular society websites, and immediate predecessors to these guideline documents (1999-2014), as referenced in current guideline documents.

Study Selection Comprehensive guideline documents including recommendations organized by class and LOE.

Data Extraction and Synthesis The number of recommendations and the distribution of LOE (A [supported by data from multiple RCTs or a single, large RCT], B [supported by data from observational studies or a single RCT], and C [supported by expert opinion only]) were determined for each guideline document.

Main Outcomes and Measures The proportion of guideline recommendations supported by evidence from multiple RCTs (LOE A).

Results Across 26 current ACC/AHA guidelines (2930 recommendations; median, 121 recommendations per guideline [25th-75th percentiles, 76-155]), 248 recommendations (8.5%) were classified as LOE A, 1465 (50.0%) as LOE B, and 1217 (41.5%) as LOE C. The median proportion of LOE A recommendations was 7.9% (25th-75th percentiles, 0.9%-15.2%). Across 25 current ESC guideline documents (3399 recommendations; median, 130 recommendations per guideline [25th-75th percentiles, 111-154]), 484 recommendations (14.2%) were classified as LOE A, 1053 (31.0%) as LOE B, and 1862 (54.8%) as LOE C. When comparing current guidelines with prior versions, the proportion of recommendations that were LOE A did not increase in either ACC/AHA (median, 9.0% [current] vs 11.7% [prior]) or ESC guidelines (median, 15.1% [current] vs 17.6% [prior]).

Conclusions and Relevance Among recommendations in major cardiovascular society guidelines, only a small percentage were supported by evidence from multiple RCTs or a single, large RCT. This pattern does not appear to have meaningfully improved from 2008 to 2018.

Introduction

In the late 1980s and early 1990s, evidence-based medicine, an approach that stresses the use of evidence from clinical research in clinical decision making, supplanted an older paradigm that valued accumulated wisdom and experience derived from unsystematic observation.1 Randomized controlled trials (RCTs), especially those that evaluate important clinical outcomes, and meta-analyses combining their results, represent the pinnacle of evidence under this framework because the randomization process attempts to equalize the distribution of unmeasured and unknown confounders, enabling investigators to compare competing treatments or strategies with the lowest risk of confounding.

Clinical guidelines for the care of patients with cardiovascular diseases, released by the American College of Cardiology/American Heart Association (ACC/AHA) and European Society of Cardiology (ESC) for more than 30 years, have integrated the evidence-based medicine framework by assigning a level of evidence (LOE) to each recommendation. Each recommendation is assigned an LOE that indicates whether the recommendation is based on multiple RCTs or a single, large RCT (LOE A), observational studies or a single RCT (LOE B), or expert opinion only (LOE C). Across a number of cardiovascular subfields, adherence to guideline recommendations translates the treatment benefits demonstrated in high-quality RCTs to improved patient outcomes.2-4

In a review of the ACC/AHA clinical practice guidelines from 2009, only 11% of recommendations were classified as LOE A.5 The authors called for greater collaboration among investigators and funders in identifying key research questions, development of streamlined clinical trial methods, and expansion of funding for clinical research. In the intervening years, some of these steps have been taken,6-8 but it is not known whether they have improved the evidence supporting cardiovascular guideline recommendations. This systematic review of current ACC/AHA and ESC guidelines and their immediate precursors was conducted to describe the evidence behind current guideline recommendations and changes in evidence over recent years.

Methods

Current ACC/AHA guidelines were identified as those posted on the ACC (https://www.acc.org/guidelines#doctype=Guidelines) and ESC (https://www.escardio.org/Guidelines/Clinical-Practice-Guidelines) websites as of February 1, 2019. Only comprehensive guideline documents were included in this systematic review; expert consensus documents, performance measures, and appropriateness criteria were not included because they do not report LOE. Focused updates were not included because they are not representative of the evidence base for an entire topic. Only guideline documents that included recommendations organized by class and LOE, clearly highlighted and separated from the rest of the text, were included for this analysis.

Current guideline documents were downloaded, and recommendations were abstracted by a single reviewer (A.C.F.) and validated by another reviewer (R.D.L). The reviewers recorded the number of recommendations included in each document, as well as the LOE (supported by data from multiple RCTs or a single, large RCT [LOE A]; supported by data from observational studies or a single RCT [LOE B]; supported by expert opinion only [LOE C]) and class for each. All recommendations are given a class that synthesizes the opinion of the guideline writing committee regarding the risks and benefits identified by the evidence and expert opinion. Class I recommendations are those for which there is evidence, general agreement, or both, that the treatment is useful or effective. Class IIa recommendations are those for which there is conflicting evidence or opinion, but the weight of evidence/opinion is in favor of the treatment’s usefulness, efficacy, or both; class IIb recommendations are those for which usefulness or efficacy is less well established. Class III recommendations are those for which there is evidence or general agreement that the treatment is not useful or effective and may be harmful.

Current guidelines were also reviewed to identify references to a previous iteration of the same guideline; for example, the 2014 AHA/ACC Guideline for the Management of Patients With Non–ST-Elevation Acute Coronary Syndromes was preceded by the 2007 ACC/AHA Guidelines for the Management of Patients With Unstable Angina/Non–ST-Elevation Myocardial Infarction.9,10 These previous iterations were then downloaded and abstracted, if they otherwise met inclusion criteria.

Recommendations in included guideline documents are clearly displayed statements highlighted and separated from the rest of the document text. Each recommendation has a clearly designated class of recommendation and LOE. Abstraction therefore involved simple reporting of the content of each document and did not require judgment on the part of the abstractors.

Data Presentation

We report the number of recommendations classified as LOE A, B, and C for each current guideline document, as well as the proportion of LOE A, B, and C recommendations by class. Because the number of recommendations included in a guideline document differ substantially by topic, we present the median proportion of recommendations with LOE A, B, and C for ACC/AHA guidelines and for ESC guidelines. We also categorized guideline documents by cardiovascular subspecialty area (electrophysiology, coronary artery disease, heart failure and myocardial disease, congenital and valvular heart disease, vascular medicine, preventive and general outpatient cardiology) and report the proportion of recommendations for each subspecialty area categorized as LOE A overall and by class of recommendation. We further report the median proportion of recommendations with LOE A, B, and C among guideline documents in each subspecialty area.

To identify changes in quality of evidence over time among current guidelines, we report the proportion of recommendations categorized as LOE A by year of guideline document release (2017-2018, 2015-2016, 2013-2014, and before 2014). We also report the proportion of recommendations categorized as LOE A in guideline documents released in the past 2 years (2017-2018) and those released before 2017. To further evaluate whether there has been a change in the evidence underpinning guideline recommendations, we compared the proportion of LOE A, B, and C recommendations in current guidelines with the proportion in prior guidelines. This procedure was performed separately for ACC/AHA and ESC guidelines. For topics that had both a current ACC/AHA and a current ESC guideline, we report the proportion of LOE A recommendations by society producing the guideline overall and by subspecialty area.

Results

Overall, LOEs from 26 current ACC/AHA guidelines (2930 recommendations) published between 2008 and 2018,10-35 and from 25 current ESC guidelines (3399 recommendations) published between 2003 and 2018, were abstracted.36-60

Across the 26 current ACC/AHA guidelines, 248 recommendations (8.5%) were classified as LOE A, 1465 (50.0%) as LOE B, and 1217 (41.5%) as LOE C (eTable 1 in the Supplement). Of 1272 class I recommendations, 180 (14.2%) were classified as LOE A. The median number of recommendations per guideline was 121 (25th-75th percentiles, 76-155), of which 7.9% (25th-75th percentiles, 0.9%-15.2%) were LOE A, 50.1% (25th-75th percentiles, 40.9%-58.4%) were LOE B, and 38.4% (25th-75th percentiles, 31.2%-49.0%) were LOE C. When guidelines were categorized by subspecialty area, the proportion of recommendations characterized as LOE A ranged from 2.0% (valvular and congenital heart disease) to 14.5% (general cardiology) (Figure 1, panel A). Overall, 1604 recommendations were classified as class I (“should do”) or class III (“should not do”). Of these recommendations, 207 (12.9%) were supported by LOE A evidence, 785 (48.9%) by LOE B evidence, and 612 (38.2%) by LOE C evidence.

Since 2015, ACC/AHA guidelines have indicated whether recommendations with LOE B were based on data from RCTs or observational studies. In the 8 guideline documents published since,11-16,34,35 there were a total of 543 recommendations with LOE B, and 144 of these recommendations (26.5%) were based on data from RCTs.

Across the 25 current ESC guidelines, 484 recommendations (14.2%) were classified as LOE A, 1053 (31.0%) as LOE B, and 1862 (54.8%) as LOE C. Of 1622 class I recommendations, 349 (21.5%) were LOE A. The median number of guideline recommendations was 130 (25th-75th percentiles, 111-154), of which 9.1% (25th-75th percentiles, 3.8%-23.1%) were LOE A, 32.6% (25th-75th percentiles, 20.5%-38.9%) were LOE B, and 50.0% (25th-75th percentiles, 40.3%-72.7%) were LOE C (eTable 2 in the Supplement). When guidelines were categorized by subspecialty area, the proportion of recommendations characterized as LOE A ranged from 2.5% (valvular and congenital heart disease) to 25.0% (general cardiology) (Figure 1, panel B). ESC guidelines contained 1884 class I and class III recommendations, 402 (21.3%) classified as LOE A, 548 (29.1%) as LOE B, and 934 (49.6%) as LOE C.

To identify trends over time, we compared evidence supporting recommendations from guideline documents written in the past 2 years (2017-2018) with that from other current guideline documents. In ACC/AHA guidelines, 5.7% of recommendations released in the past 2 years were supported by LOE A evidence compared with 9.5% of recommendations released earlier (eFigure in the Supplement). In ESC guidelines, 17.4% of recommendations released in the past 2 years were supported by LOE A evidence, compared with 12.8% of recommendations released earlier.

Changes From Prior to Current Guidelines

ACC/AHA Guidelines

Sixteen current ACC/AHA guideline documents, published between 2008 and 2018, had a prior document for comparison.10,12,14-17,19,21,23-28,33,34 These 16 documents contained a total of 2159 recommendations (median, 130). The 16 prior documents were published between 1999 and 2013 and included 3154 recommendations (median, 135).9,59,61-74

Among the 16 current ACC/AHA guideline documents with available prior guideline documents, the median proportion of LOE A recommendations was 9.0% (25th-75th percentiles, 4.8%-15.1%); the median proportion of LOE A recommendations in prior guidelines was 11.7% (25th-75th percentiles, 5.8%-17.5%) (Figure 2, panel A). Among class I recommendations, there was similarly no meaningful change in the proportion that were LOE A between current and prior guidelines (median, 16.1% in current vs 20.7% in prior). By contrast, when comparing the proportion of LOE B and C recommendations in prior ACC/AHA guidelines and current guidelines, the proportion of LOE B recommendations increased (median, 41.9% vs 51.0%) and the proportion of LOE C recommendations decreased (median, 51.9% vs 36.7%).

When looking at number rather than proportion of LOE A recommendations, findings were similar: The median number of LOE A recommendations in prior guidelines was 10.5 (25th-75th percentiles, 7.5-28.0) compared with 11.5 (25th-75th percentiles, 6.5-16.0) in current guidelines, and in 10 of 16 current/prior guideline dyads, there were more LOE A recommendations in the prior iteration of the guidelines.

ESC Guidelines

Sixteen current ESC guideline documents, published between 2014 and 2018, had a corresponding prior guideline document for comparison.36-51 These 16 documents included 2447 recommendations (median, 142). The 16 prior documents were published between 2004 and 2014 and included 2112 recommendations (median, 110).42,64,75-88

Among the 16 current ESC guideline documents with available prior guideline documents, the median proportion of LOE A recommendations was 15.1% (25th-75th percentiles, 3.7%-26.4%); the median proportion of LOE A recommendations among prior guidelines was 17.6% (25th-75th percentiles, 2.4%-27.6%) (Figure 2, panel B). Among class I recommendations, there was similarly little change in the proportion that were LOE A between current and prior guidelines (median, 26.9% in current vs 23.6% in prior). There were similarly small differences between current and prior guidelines in the proportion of recommendations that were LOE B (median, 31.7% [current] vs 33.4% [prior]) or LOE C (median, 49.3% [current] vs 48.0% [prior]).

When looking at number of recommendations, the median number of LOE A recommendations in current guideline documents was 21 (25th-75th percentiles, 4.7-37.2), compared with 26 (25th-75th percentiles, 2.5-35.2) in prior guideline documents.

Comparison Between ACC/AHA and ESC Guidelines

Among current guideline documents, 17 topics had both a current ESC and current ACC/AHA version. Among these guideline documents, 13.3% of recommendations in ESC guidelines and 7.5% of recommendations in ACC/AHA guidelines were supported by LOE A evidence. In 13 cases, the ESC guideline document had a greater proportion of recommendations supported by LOE A evidence. When guideline documents were grouped by subspecialty, ESC guidelines included a greater proportion of recommendations classified as LOE A in every subspecialty area except valvular and congenital heart disease (Figure 3). A greater proportion of recommendations in ESC guidelines, both current and prior, were class I, LOE A than in ACC/AHA guidelines (Figure 4).

Discussion

In this review of evidence supporting major society cardiovascular guidelines, less than 10% of recommendations from current ACC/AHA guidelines and less than 15% of recommendations from current ESC guidelines were supported by evidence from multiple high-quality RCTs and characterized as LOE A; approximately 80% of strong (class I or III) recommendations were not characterized as LOE A. Furthermore, there was wide variety by subject area—in some guideline documents, more than 33% of recommendations were characterized as LOE A, but others had no LOE A recommendations at all. Among guideline documents that have been updated, there was no meaningful change in the proportion or number of recommendations characterized as LOE A from prior to current guidelines. Taken together, these results demonstrate that efforts over the past decade to simplify and facilitate clinical trials have not yet translated into an evidence base better supported by RCTs.

In 2009, Tricoci et al5 analyzed data from ACC/AHA guidelines published from 1984 to 2008. At that time, 12% of guideline recommendations were classified as LOE A, including 19% of class I recommendations. Because LOEs were included in guidelines starting in 1998, Tricoci et al were only able to assess changes in LOE over time in 6 guideline documents, but in these 6 guideline documents, they found minimal increases in LOE A recommendations compared with LOE B and C. In follow-up to this effort, the evidence underlying guideline recommendations in other medical and surgical subspecialties was systematically reviewed in a number of studies. Overall, very few recommendations were supported by high-quality evidence from multiple RCTs.89-99

In the article by Tricoci et al, the authors pointed out many of the flaws in the clinical trial enterprise, including inefficiency, fragmentation, and reliance on industry funding, resulting in narrowly focused trials in highly selected populations, designed to achieve regulatory approval but not necessarily to provide useful evidence for patients, clinicians, and payers. The authors called for increased funding for practical clinical trials evaluating the comparative effectiveness of existing products, increased collaboration in setting a research agenda, and novel methods of conducting clinical trials with less waste. Over the last decade, public-private partnerships have been developed to fund clinical trials asking patient-centered questions,7 trial designs leveraging administrative data and existing registries to capture baseline characteristics and long-term outcomes have been deployed in service to these questions,6,100,101 and a series of meetings have brought leaders in various cardiovascular subfields together to collaboratively devise research agendas.102,103

The present study shows that, despite these efforts, the proportion of guideline recommendations supported by high-quality evidence did not increase. When directly comparing current guideline documents with prior documents covering the same topic there was similarly not a meaningful increase in recommendations with LOE A classification from the prior version to the current version of guidelines in either the ACC/AHA or ESC guidelines. Although it is possible that RCTs both convincingly answer a single question (leading to a single LOE A recommendation) and raise new questions (leading to multiple non–LOE A recommendations), the absolute number of LOE A recommendations did not meaningfully change from prior to current guideline iterations. In both the ACC/AHA and the ESC guidelines, the large majority of patient care recommendations were based on nonrandomized evidence, even class I (“should do”) and class III (“should not do”) recommendations. The lack of RCT evidence supporting most recommendations in the guidelines was compounded by variability among subspecialties within cardiovascular medicine; some subspecialty guidelines contained almost no LOE A recommendations. Although the ESC and ACC/AHA guidelines use similar evidence to generate recommendations, a greater proportion of recommendations overall in the ESC guidelines were classified as LOE A, highlighting differences in the way that these professional societies interpret data and make guideline recommendations, and/or hesitancy of guideline writing committees to categorize as LOE A recommendations based on evidence from RCTs that enrolled patients entirely in other regions of the world.

In the ACC/AHA guidelines, there was a small increase in the proportion of recommendations that were LOE B and a decrease in the proportion that were LOE C when comparing current and prior guideline documents. LOE B recommendations are supported either by observational studies or single RCTs, so this increase in LOE B recommendations could indicate that the proportion of recommendations supported by randomized evidence is increasing; however, relatively few LOE B recommendations were supported by randomized evidence. More likely, the increase in LOE B recommendations from prior to current ACC/AHA guideline documents can be explained, at least in part, by the elaboration of new observational “big data” sources and application of advanced statistical methods, which have led researchers to ask and answer questions using observational study designs.104 Although evidence generated from such studies is valuable in many circumstances, comparative effectiveness analyses using observational data are limited by residual confounding, and in most circumstances a well-conducted RCT is the only study design that enables a true comparison (and cause-effect relationship assessment) between 2 medications, procedures, or treatment strategies.104 In a review of cardiology guidelines, 19% of class I recommendations supported by 1 RCT or observational evidence only, and more than 25% of class I recommendations supported by expert opinion only, were downgraded or reversed in the next edition of the guidelines, compared with less than 10% of recommendations supported by higher-quality evidence.105

Solid RCT evidence delineates treatments and strategies that lead to better patient outcomes, which can then be implemented in clinical practice.106 By contrast, in the absence of RCT evidence, the association between clinical practices and outcomes is less certain. The decline in cardiovascular mortality has decelerated over the past several years.107 Efforts to bolster the evidence base—eg, pragmatic clinical trials, registry-based clinical trials, and clinical trials conducted within health systems—may help forestall this trend.

Limitations

This study has several limitations. First, the evidence supporting major society guideline recommendations is a surrogate for the totality of the evidence in cardiology, rather than a direct measurement. The quality of evidence supporting each recommendation was not independently assessed, and it is possible that an increasing proportion of LOE B recommendations are supported by RCT evidence that is insufficient to characterize as LOE A. Similarly, the proportion of guideline recommendations supported by LOE A evidence may not be a perfect surrogate for the totality of evidence, because advances in the field may make prior guideline recommendations obsolete and thus removed from the next edition of the guidelines. Alternatively, standards for LOE A designation may have changed over time.

Second, exclusion of focused updates, which are usually undertaken when new RCT evidence is generated, may lead to underestimation of the proportion of LOE A recommendations; however, this limitation should not affect findings regarding changes in LOE over time.

Third, this review describes only evidence supporting cardiology guidelines, rather than across medical and surgical subspecialties; however, prior studies have shown a low level of high-quality evidence in other fields. Moreover, the high global prevalence of cardiovascular disease suggests that efforts to build a higher-quality cardiovascular medicine evidence base might have a large effect on global health.

Conclusions

Among recommendations in major cardiovascular society guidelines, only a small percentage were supported by evidence from multiple RCTs or a single, large RCT. This pattern does not appear to have meaningfully improved from 2008 to 2018.

Back to top Article Information

Corresponding Author: Renato D. Lopes, MD, PhD, MHS, Duke Clinical Research Institute, PO Box 17969, Durham, NC 27715 (renato.lopes@duke.edu).

Accepted for Publication: February 12, 2019.

Author Contributions: Drs Fanaroff and Lopes had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Fanaroff, Smith, Lopes.

Acquisition, analysis, or interpretation of data: Fanaroff, Califf, Windecker, Lopes.

Drafting of the manuscript: Fanaroff, Lopes.

Critical revision of the manuscript for important intellectual content: Fanaroff, Califf, Windecker, Smith, Lopes.

Statistical analysis: Fanaroff, Lopes.

Obtained funding: Lopes.

Administrative, technical, or material support: Califf, Lopes.

Supervision: Lopes.

Conflict of Interest Disclosures: Dr Fanaroff reported support from a career development grant from the American Heart Association (17FTF33661087). Dr Califf reported serving as the Commissioner of Food and Drugs for the US Food and Drug Administration from February 2016 to January 2017; serving as Deputy Commissioner for Medical Products and Tobacco for the US Food and Drug Administration from February 2015 to January 2016; serving on the corporate board for Cytokinetics and as board chair for the People-Centered Research Foundation; and receiving consulting fees from Merck, Biogen, Genentech, Eli Lilly, and Boehringer Ingelheim. Dr Windecker reported research and educational grants to his institution from Abbott, Amgen, Bayer, Boston Scientific, Biotronik, Edwards Lifesciences, Medtronic, St Jude, and Terumo. Dr Lopes reported receiving research grants from Bristol-Myers Squibb, GlaxoSmithKline, Medtronic, Pfizer, and Sanofi and receiving personal fees from Amgen, Bayer, Boehringer Ingelheim, Bristol-Myers Squibb, GlaxoSmithKline, Medtronic, Pfizer, and Sanofi. Dr Smith reported no disclosures.