Abstract

Importance Many patients and physicians assume that the safety and effectiveness of newly approved therapeutic agents is well understood; however, the strength of the clinical trial evidence supporting approval decisions by the US Food and Drug Administration (FDA) has not been evaluated.

Objectives To characterize pivotal efficacy trials (clinical trials that serve as the basis of FDA approval) for newly approved novel therapeutic agents.

Design and Setting Cross-sectional analysis using publicly available FDA documents for all novel therapeutic agents approved between 2005 and 2012.

Main Outcomes and Measures Pivotal efficacy trials were classified according to the following design features: randomization, blinding, comparator, and trial end point. Surrogate outcomes were defined as any end point using a biomarker expected to predict clinical benefit. The number of patients, trial duration, and trial completion rates were also determined.

Results Between 2005 and 2012, the FDA approved 188 novel therapeutic agents for 206 indications on the basis of 448 pivotal efficacy trials. The median number of pivotal trials per indication was 2 (interquartile range, 1-2.5), although 74 indications (36.8%) were approved on the basis of a single pivotal trial. Nearly all trials were randomized (89.3% [95% CI, 86.4%-92.2%]), double-blinded (79.5% [95% CI, 75.7%-83.2%]), and used either an active or placebo comparator (87.1% [95% CI, 83.9%-90.2%]). The median number of patients enrolled per indication among all pivotal trials was 760 (interquartile range, 270-1550). At least 1 pivotal trial with a duration of 6 months or greater supported the approval of 68 indications (33.8% [95% CI, 27.2%-40.4%]). Pivotal trials using surrogate end points as their primary outcome formed the exclusive basis of approval for 91 indications (45.3% [95% CI, 38.3%-52.2%]), clinical outcomes for 67 (33.3% [95% CI, 26.8%-39.9%]), and clinical scales for 36 (17.9% [95% CI, 12.6%-23.3%]). Trial features differed by therapeutic and indication characteristics, such as therapeutic area, expected length of treatment, orphan status, and accelerated approval.

Conclusions and Relevance The quality of clinical trial evidence used by the FDA as the basis for recent approvals of novel therapeutic agents varied widely across indications. This variation has important implications for patients and physicians as they make decisions about the use of newly approved therapeutic agents.

The approval of a drug by the US Food and Drug Administration (FDA) conveys that the product is safe and effective. An Internet-based survey of a national probability sample of 4316 US adults (2944 respondents [68% response rate]) found that 39% report believing that the FDA approves only “extremely effective” drugs and 25% only drugs without serious adverse effects.1 Some physicians make similar assumptions about effectiveness and safety, expecting that patients are likely to benefit from newly approved therapies.2-5

FDA review of new drug applications is guided by the Federal Food, Drug, and Cosmetic Act, which requires “adequate and well controlled investigations” to determine efficacy.6 FDA guidance suggests that drug manufacturers submit at least 2 trials, each providing independent evidence of efficacy—such studies are known as “pivotal” efficacy trials—but also implies flexibility, describing circumstances in which a single efficacy trial might be sufficient to support approval.7 Moreover, for certain applications, the FDA provides written guidance on the design of pivotal efficacy trials, including features of trial design, such as sample selection and choice of comparator,8-10 and may provide further guidance in meetings with individual sponsors.11 As an example, for therapeutic agents evaluated through the accelerated approval pathway, which aims to speed approval of therapeutic agents that treat life-threatening diseases, the FDA permits pivotal efficacy trials to use surrogate end points that are “reasonably likely” to predict clinical benefit.12

The clinical research findings available at the time of a drug’s approval have important implications: if made public, these findings represent the only source of information available to patients and their physicians as they decide whether to use a newly approved drug. However, flexible approval standards may lead to some therapeutic agents being approved by the FDA on the basis of numerous rigorously designed clinical trials and others on the basis of fewer or less robust studies, leading to differing levels of certainty about the risks and benefits of newly approved drugs. Accordingly, we sought to systematically examine this issue, evaluating the strength of the clinical trial evidence supporting FDA approval decisions for novel therapeutic agents—pharmacologics and biologics—between 2005 and 2012 by characterizing key features of pivotal efficacy trials, such as trial size, design, duration, and end points.

Methods

Data Sources

Drugs@FDA is a publicly accessible database available through the FDA’s website that lists regulatory actions, such as approvals and drug labeling changes, for all currently approved prescription therapeutic agents. Records for each approved agent are hyperlinked to FDA medical reviews, which are lengthy documents that outline the clinical evidence used to establish the efficacy and safety of the novel agent prior to approval. The Drugs@FDA database was downloaded on January 11, 2012, and on May 1, 2013. Medical reviews were accessed several times between January 2012 and June 2013.

Study Sample

We constructed a sample of novel therapeutic agents (ie, new molecular entities or novel biologic drugs) first approved by the FDA between January 1, 2005, and December 31, 2012, excluding generic drugs, reformulations, and combination therapies of nonnovel therapeutic agents (Figure). We also excluded nontherapeutic agents, such as diagnostic and contrast agents (eAppendix 1 in Supplement), and removed any duplicate records.

Therapeutic Agent and Indication Characteristics

Using information provided within the Drugs@FDA database, we categorized each novel therapeutic agent by year of approval and as a pharmacologic entity (ie, small molecule) or biologic.13 Additionally, agents were classified by orphan status, a designation made by the FDA that affords extended market exclusivity for drugs that treat rare diseases (the Drugs@FDA database only indicates orphan status for biologics approved after 2010). Using FDA approval letters, which are also hyperlinked in the Drugs@FDA database, we identified therapeutic agents approved through the accelerated approval pathway and the indication for which all novel therapeutic agents were initially approved for use. Subsequently, indications were categorized by expected length of treatment: acute, intermediate, or chronic. The expected length of use was less than 1 month for acute treatments, between 1 month and 2 years for intermediate treatments, and greater than 2 years for chronic treatments (eAppendix 2 in Supplement). Additionally, we used the World Health Organization’s Anatomic Therapeutic Classification system, contextualized for clinical relevance, to categorize each indication into 1 of 8 therapeutic areas.14 Last, 2 investigators (N.S.D., J.S.R.) determined the total number of patients exposed to the novel therapeutic agent during clinical development, ie, the total safety population (eAppendix 3 in Supplement).

Identification of Pivotal Efficacy Trials

For each novel therapeutic agent, 1 investigator (J.A.A.) identified the pivotal efficacy trials used as the basis for approval. Generally, these trials are labeled in FDA medical reviews as “pivotal,” and their design and findings are discussed in detail. For approvals in which no trial was explicitly labeled “pivotal” in the FDA medical review, we identified trials described as essential to approval or those prioritized within the review using criteria such as substantial discussion of study design (ie, inclusion and exclusion criteria, thorough description of study protocol) and independent analysis of results (ie, not pooled with other studies). Additionally, any efficacy trial reviewed as part of a resubmitted application was considered pivotal to approval. Two investigators (N.S.D., J.S.R.) subsequently validated identification of all pivotal efficacy trials through independent review, resolving conflicts by consensus.

Pivotal Efficacy Trial Features

Each pivotal trial was categorized according to its use of randomization and blinding based on the FDA reviewer’s description of the trial. In addition, we also recorded the type of comparator, primary trial end point(s), the number of treated patients (overall and intervention group), trial duration, and completion rate (eAppendix 3 in Supplement). Completion rate was calculated by dividing the number of patients completing the trial by the number of treated patients (overall and intervention group). Primary trial end points were classified as clinical outcomes, clinical scales, or surrogate outcomes based on an established framework and a recent Institute of Medicine report (eAppendix 3 in Supplement).15,16 Clinical outcomes, such as death or rate of hospitalization, measure patient survival or function. Clinical scales, such as the Crohn Disease Activity Index or the visual analog scale for pain, represent rubrics for the quantification of subjective patient-reported symptoms. Surrogate outcomes, such as levels of glycated hemoglobin or hepatitis C ribonucleic acid, represent biomarkers expected to predict clinical benefit. The initial abstraction was performed by 1 investigator (J.A.A.). Two investigators (N.S.D., J.S.R.) subsequently validated characterization of all pivotal trials through independent review and abstraction, resolving conflicts by consensus.

Statistical Analysis

Using descriptive statistics, we characterized the novel therapeutic agents included in our sample and the indications for which they were initially approved for use. Next, we used descriptive statistics to characterize features across the overall sample of pivotal efficacy trials as well as the features of these trials aggregated at the indication level, ie, the summary of all pivotal efficacy trials used to support the approval of each indication. We then used χ2, Wilcoxon, and Kruskal-Wallis tests as appropriate to examine differences among novel therapeutic agent and indication characteristics, including therapeutic area, expected length of therapy, agent type, orphan status, and accelerated approval, all of which were preplanned prior to data collection.

Analyses were performed using Microsoft Excel 2010 and JMP version 7.0.1 (SAS Institute Inc). All statistical tests were 2-tailed and used a type I error rate of .01 to account for multiple comparisons across 5 therapeutic agent and indication characteristics.

Results

Between 2005 and 2012, the FDA approved 188 novel therapeutic agents: 154 (81.9%) were pharmacologics and 34 (18.1%) were biologics. The FDA had granted orphan status to 31 (16.5%), and 22 (11.7%) were approved through the accelerated approval pathway (Table 1). These 188 novel agents were approved for use for 206 indications: 171 (91.0%) for a single indication, 16 (8.5%) for 2 indications, and 1 (0.5%) for 3 indications. More than half (108 [52.4%]) of indications required chronic treatment, 58 (28.2%) required intermediate-length treatment, and 40 (19.4%) required acute treatment. Three therapeutic areas accounted for nearly half of indications: 41 agents (19.9%) were used to treat cancer, 29 (14.1%) to treat infectious disease, and 23 (11.2%) to treat cardiovascular disease, diabetes mellitus, or hyperlipidemia. The median safety population, the total number of patients exposed to the novel therapeutic agent during clinical development, was 1143 (interquartile range [IQR], 503-2600).

Pivotal Efficacy Trial Features

Pivotal efficacy trials were identified for 201 of 206 indications (Figure); 4 novel therapeutic agents were approved (1 for 2 indications) without a pivotal efficacy trial. A total of 448 pivotal trials were identified: 283 (63.2%) were explicitly labeled “pivotal,” and 165 (36.8%) were inferred as pivotal based on the criteria described previously. The vast majority of pivotal trials were randomized (400 [ 89.3% {95% CI, 86.4%-92.2%}]) and double-blinded (356 [79.5% {95% CI, 75.7%-83.2%}]) (Table 2). More than half of the trials used a placebo comparator (247 [55.1% {95% CI, 50.5%-59.8%}]), 143 (31.9% [95% CI, 27.6%-36.3%]) used an active comparator, and 58 (12.9% [95% CI, 9.8%-16.1%]) had no comparator. The primary end point was a surrogate outcome for 219 (48.9% [95% CI, 44.2%-53.5%]) trials, a clinical outcome for 130 (29.0% [95% CI, 24.8%-33.2%]) trials, and a clinical scale for 99 (22.1% [95% CI, 18.2%-26.0%]) trials. Median total and intervention group patient populations were 446 (IQR, 205-678) and 271 (IQR, 133-426). Median trial duration was 14.0 weeks (IQR, 6.0-26.0 weeks); 113 trials (25.2% [95% CI, 21.2%-29.3%]) lasted 6 months or longer (Table 3). The median completion rate was 86.6% (IQR, 77.9%-93.1%).

Trial Features by Therapeutic Agent and Indication Characteristics

Features of pivotal efficacy trials differed by therapeutic agent and indication characteristics. Trials of therapeutic agents used for cancer were least likely to be randomized (47.3% [95% CI, 33.7%-60.9%] vs 95.2% [95% CI, 93.0%-97.3%]; P < .001) and double-blinded (27.3% [95% CI, 15.1%-39.4%] vs 86.8%[ 95% CI, 83.4%-90.1%]; P < .001) (Table 2). An active comparator was used more frequently in trials of therapeutic agents approved for infectious disease indications than in trials of agents approved for other indications (68.4% [95% CI, 56.0%-80.9%] vs 26.6% [95% CI, 22.2%-31.0%]; P < .001). Clinical outcomes and scales were infrequently used in trials of agents approved for cancer (9 [16.4% {95% CI, 6.3%-26.5%}]) and cardiovascular disease, diabetes mellitus, and hyperlipidemia (11 [15.1% {95% CI, 6.7%-23.5%}]). Surrogate end points were used in nearly all trials of agents approved through the accelerated approval pathway (38 [95.0% {95% CI, 87.9%-100%}]), in contrast to fewer than half (181 [44.4% {95% CI, 39.5%-49.2%}]) of trials among therapeutic agents receiving nonaccelerated approval.

The median patient population in the intervention group was smaller among therapeutic agents with orphan status when compared with those without (98 [IQR, 53-184] vs 294 [IQR, 157-454]; P < .001) and in trials of agents approved via the accelerated approval pathway when compared with agents approved via nonaccelerated pathway (142 [IQR, 78-260] vs 289 [IQR, 142-446]; P < .001) (Table 3). Trial duration varied according to expected length of treatment (P < .001) and was shorter for pharmacologics when compared with biologics (12.0 [IQR, 6.0-24.0] weeks vs 24.0 [IQR, 18.0-49.6] weeks; P < .001).

Aggregated Pivotal Efficacy Trial Features Supporting Approved Indications

Among 201 indications, the median number of trials per indication was 2 (IQR, 1-2.5) (Table 4); 74 indications (36.8%) were approved on the basis of a single trial, 77 (38.3%) on 2, and 50 (24.9%) on 3 or more. Among the aggregated pivotal efficacy trials supporting these indications, median total and intervention group patient populations were 760 (IQR, 270-1550) and 445 (IQR, 169-936). Although at least 1 trial using clinical outcomes or clinical scales supported the approval of 73 indications (36.3% [95% CI, 29.6%-43.0%]) and 39 indications (19.4% [95% CI, 13.9%-24.9%]), respectively (Table 5), trials using clinical outcomes or clinical scales formed the exclusive basis of approval for slightly fewer indications: 67 (33.3% [95% CI, 26.8%-39.9%]) and 36 (17.9% [95% CI, 12.6%-23.3%]), respectively. In addition, trials using surrogate end points as their primary outcome formed the exclusive basis of approval for 91 indications (45.3% [95% CI, 38.3%-52.2%]). At least 1 trial of at least 6 months’ duration supported the approval of 68 indications (33.8% [95% CI, 27.2%-40.4%]).

Aggregated Trial Features Supporting Approved Indications by Therapeutic Agent and Indication Characteristics

The features of the aggregated pivotal efficacy trials supporting approved indications differed by therapeutic agent and indication characteristics. Most therapeutic agents approved for cancer indications were approved on the basis of a single trial, whereas the approval of therapeutic agents for cardiovascular disease, diabetes mellitus, or hyperlipidemia and for psychiatric indications often relied on at least 3 trials (Tables 4 and 5). Median numbers of overall and intervention group patients were larger among aggregated trials supporting indications within these therapeutic areas.

There was no difference in the proportion of indications approved through the accelerated approval pathway on the basis of multiple trials when compared with nonaccelerated approval indications (56.5% [95% CI, 34.6%-78.4%] vs 64.0% [95% CI, 56.9%-71.2%]; P = .48). More therapeutic agents indicated for chronic treatment were supported by at least 1 trial of 6 months’ duration when compared with agents indicated for acute or intermediate-length treatment (44.4% [95% CI, 34.9%-54.0%] vs 2.8% [95% CI, 0%-8.4%] and 33.3% [95% CI, 20.7%-46.0%], respectively; P < .001).

Discussion

Our characterization of pivotal efficacy trials—trials that serve as the basis of FDA approval—for all novel therapeutic agents approved between 2005 and 2012 demonstrates that the quality of clinical trial evidence used by the FDA to make approval decisions varied widely across indications. Although the vast majority of indications were supported by at least 1 randomized, double-blinded trial, there was wide variation in trials’ choice of comparators and end points, duration, size, and completion rate. In addition, just more than one-third of indications were approved on the basis of a single pivotal efficacy trial.

The variation in the quality of clinical trial evidence used by the FDA to assess the efficacy of novel therapeutic agents highlights the agency’s flexible standards for approval. Such regulatory flexibility allows for a customized approach to approval, including the ability to rapidly approve potentially effective therapies for life-threatening diseases, such as certain cancers, or those diseases for which there is no existing effective treatment, such as orphan diseases. These approvals can be made without requiring costly and time-consuming randomized, double-blinded, controlled trials, although these trials are regarded as the gold standard for evaluation.17-19 Indeed, the FDA has provided guidance on approaches to accelerate clinical development of novel therapeutic agents8-10,20 and has cited its willingness to rapidly approve new drugs in recent year-end reviews of drug approvals.21,22 Substantial variation has been described among pivotal efficacy trials supporting the approval of cancer drugs,23 and this flexibility may well be warranted given the limited number of effective therapies and the poor prognosis associated with cancer.

Understanding the strength of clinical trial evidence of newly approved therapeutic agents has important implications for patients and physicians. When medications become available on the market, decisions must be made about their use, likely informed by how well safety and effectiveness are understood. Comparative effectiveness information, which is not required as part of FDA approval and involves comparison of an intervention with an active control, was available for less than half of indications, consistent with prior research,24 but leaving uncertainty about the benefits and safety of these medications when compared with other available therapeutic agents. Similarly, although patient-important clinical outcomes and scales were used in many pivotal trials, trials using surrogate end points as their primary outcome formed the exclusive basis of approval for nearly half of the approved indications in our study. This reliance on surrogate outcomes leaves patients and physicians to extrapolate clinical benefits from trials, again raising questions about the certainty of the medications’ benefits in practice.25 Last, we found that the majority of trials evaluating therapeutic agents indicated for chronic treatment lasted less than 1 year, raising questions about the certainty of these medications’ long-term efficacy and safety.

Because comprehensive safety evaluations are difficult to undertake as part of randomized controlled trials, particularly smaller trials, these findings clarify the importance of adopting a “life-cycle” approach, both for drug safety and for improved understanding of drug effectiveness. In 2006, the Institute of Medicine Report on the Future of Drug Safety recommended that the FDA monitor and evaluate the benefits and risks of drug therapies not only prior to their approval but throughout their entire market life.26 This so-called life-cycle approach suggests that new information on the benefits and safety of therapeutic agents should be continually collected. It also requires adequate and robust postmarket surveillance systems that allow reassessments of drug efficacy and safety after market introduction.

However, communicating this updated information to patients and physicians is critical. A recent Institute of Medicine committee report similarly recommended that the FDA implement a benefit and risk assessment and management plan that would summarize the FDA’s evaluation of a drug’s benefit-risk profile in a single document and that would be continuously updated during the entire market life of the product.27,28 Alternatively, or as part of this effort, the FDA could provide a summative statement, or even a grade, for each approval to signal the quality of clinical trial evidence used to determine safety and efficacy, allowing therapeutic agents approved on the basis of more robust evidence to be distinguished from those approved on the basis of less robust evidence. Just as the FDA has publicly declared its intention to encourage investigators to use innovative trial designs that are as effective as standard designs but less burdensome and time-consuming and to identify qualifying biomarkers that accurately predict outcomes to make clinical trials more efficient,29 it also must ensure that patients and physicians understand how to interpret the results of these trials and the likelihood of experiencing benefit or harm when deciding to use these newly approved agents.

Our study has several limitations. Although pivotal efficacy trials represent the primary source of information about novel therapeutic efficacy at approval, other trials that provide supplementary efficacy data are discussed transiently in FDA medical reviews but were not systematically assessed. Although the FDA is more likely to first approve novel therapeutic agents for use,30 the agency also may rely on information from use of the drugs in other countries when evaluating these agents for approval, and we did not assess this information. Additionally, efficacy represents just one component of FDA review, which also covers safety, pharmacology, chemistry, and manufacturing. Last, our study was limited to the approval of new molecular entities and novel biologics; however, because reformulated and generic drugs can be approved on the basis of bioequivalence studies, it is likely that our study captured the majority of pivotal efficacy trials used for evaluations of novel therapeutic agents. Moreover, our findings are consistent with a prior study of high-risk cardiovascular devices, which found many to be approved on the basis of trials that lack adequate strength and may be prone to bias.31

Conclusions

The quality of clinical trial evidence used by the FDA as the basis of approvals of novel therapeutic agents between 2005 and 2012 varied widely across indications. This variation has important implications for patients and physicians as they make decisions about the use of newly approved therapeutic agents and has the potential to inform current FDA regulatory approval standards and postmarket surveillance initiatives.

Back to top Article Information

Corresponding Author: Joseph S. Ross, MD, MHS, Section of General Internal Medicine, Yale University School of Medicine, PO Box 208093, New Haven, CT 06520 (joseph.ross@yale.edu).

Author Contributions: Mr Downing and Dr Ross had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Downing, Ross.

Acquisition of data: Downing, Aminawung, Ross.

Analysis and interpretation of data: All authors.

Drafting of the manuscript: Downing, Aminawung, Ross.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Downing, Aminawung, Ross.

Obtained funding: Downing, Ross.

Administrative, technical, or material support: Downing, Aminawung.

Study supervision: Ross.

Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Dr Shah reported receiving travel expenses from IMS Health. Drs Krumholz and Ross reported receiving support from Medtronic Inc to develop methods of clinical trial data sharing, from the Centers of Medicare and Medicaid Services (CMS) to develop and maintain performance measures used for public reporting, and from the US Food and Drug Administration (FDA) to develop methods for postmarket surveillance of medical devices. Dr Krumholz reported that he chairs a scientific advisory board for UnitedHealthcare. Dr. Ross reported serving as a member of a scientific advisory board for FAIR Health Inc. No other authors reported disclosures.

Funding/Support: Support for this study was provided in part by The Pew Charitable Trusts. Dr Krumholz is supported by a National Heart, Lung, and Blood Institute Cardiovascular Outcomes Center Award (1U01HL105270-02). Dr Ross is supported by the National Institute on Aging (K08 AG032886) and by the American Federation for Aging Research through the Paul B. Beeson Career Development Award Program.

Role of the Sponsor: The Pew Charitable Trusts had no role in the design and conduct of the study; the collection, management, analysis, and interpretation of the data; the preparation, review, or approval of the manuscript; or the decision to submit the manuscript for publication.