We identified 27 studies that compared registered and published trial outcomes. Among these studies, the proportion of discrepant primary outcomes between registration and publication was highly variable, with four studies observing outcome changes in more than 50 % of trials, and eight studies observing outcomes changes in fewer than 20 %. The frequency with which discrepancies were identified did not appear to vary based on funding source, journal type, or when trials were published, though our ability to detect potential differences was limited by relatively small sample sizes and unstandardized methods across the included studies. Most of the studies identified were similar in regard to the time frame analyzed and the journals and clinical topics studied. Additionally, nearly all of the included studies used a consistent method of categorizing outcome differences, based on the scheme established by Chan et al. [10]. One important source of the variability in reported results is that most studies compared published outcomes to the outcome listed in the registry at the time of the search, without attempting to identify the outcome registered at the inception of the study. For the six studies that compared published outcomes with the outcomes registered prior to the initiation of patient enrollment, the frequency of discrepancies were more similar to one another and higher than for the remaining 21 studies. Because the comparison with the outcome prior to patient enrollment is the most relevant for those attempting to understand the impact of outcome switching on the validity of results of clinical trials, the median value from these six studies (41 %, IQR 33–48 %) provides an informative estimate for characterizing the extent of this problem.

This finding supports previous observations that registry entries are often either initiated or modified after trial completion [7, 22]. Additionally, it suggests that for surveillance studies of registry practices to be meaningful and reproducible, authors should utilize best practices to improve study validity. These include assessing changes to registry entries over time, establishing clear definitions for discrepant outcomes, and utilizing multiple independent raters to compare registered and published outcomes.

While clinical trial registries and recent legislative efforts mandating their use have the potential to improve registry utilization by trial sponsors and investigators, these measures do little good unless reviewers, journal editors, and educators utilize them. Recent evidence suggests that reviewers do not routinely use trials registries when assessing trial manuscripts [39]. In some cases there are good reasons to change study outcomes or other methodological details while a trial is ongoing, though it is critical that published manuscripts disclose and explain these changes. Other times, outcome changes may represent efforts by investigators to spin favorable conclusions from their data. In either case, reviewers and editors can and should use registry data to ensure that appropriate outcome changes are transparent and that inappropriate changes are prevented.

All but one of the studies included clinical trials that were initiated before 2005, which was when the ICMJE policy mandating prospective trial registration as a condition of publication in member journals went into effect. Registry utilization has increased dramatically since that time, but the quality of registry data remains inconsistent [22, 24]. The single study we examined that was limited to trials initiated after the ICMJE statement still found primary outcome inconsistencies in 15 of 55 (27 %) included trials [16]. Importantly, we did not find evidence of a consistent difference in registered and published outcomes according to trial funding source. As a result, any measures taken to improve the consistency of outcome reporting should target both industry-funded and non-industry-funded trials. As registry utilization continues to evolve, it is unclear whether outcome consistency will improve or whether inconsistencies will remain common. It is likely that this depends in part on the willingness of both journal editors and reviewers to demand that authors are held accountable to registered protocols when publishing results. Unfortunately, very few journals currently prioritize the assessment of registry entries during the peer review process [40].

Since 2008, in addition to facilitating the assessment of methodological consistency, Clinicaltrials.gov has included a results database that allows the registry to serves as a publicly available source of trial outcome data. While utilization of the results database on the part of investigators has been mixed, improved compliance with this feature has the potential to streamline comparisons between planned and completed outcome analyses [41]. Future surveillance studies will be needed to assess improvements in outcome consistency that might result from new policies, greater familiarity of investigators and sponsors with the registry process, and greater scrutiny of registries by editors and reviewers.

Our findings are consistent with those reported in a 2011 Cochrane Review of studies comparing trial protocols with published trial manuscripts [42]. This Cochrane Review identified three studies, all also included in our analysis, that compared registered primary outcomes to published outcomes, with observed primary outcome discrepancy rates between 18 % and 31 %. Additionally, the Cochrane Review identified three studies that compared the consistency between outcomes identified in study protocols and those in published reports, finding discrepancies in 33–67 % of included trials. As with our review, the presence of heterogeneous trial populations and methods prevented the Cochrane authors from performing a meta-analysis, though they observed that discrepancies between study protocols and published manuscripts commonly involved other important aspects of trial design in addition to outcome definitions, including eligibility criteria, sample size goals, planned subgroup analyses, and methods of analysis.

This systematic review has several limitations that should be considered when interpreting these results. First, studies involving outcome discrepancies are challenging to capture via a database search, because relevant studies are not limited to a single disease process or intervention, and many of the key words relevant to this topic are commonly included in unrelated abstracts (e.g., “bias,” “primary outcome,” “clinicaltrials.gov,” “randomized controlled trial”). We addressed this issue by using an intentionally broad search strategy, and by reviewing the reference lists of all included studies. Despite this, we may have missed some relevant publications. Second, the studies identified were heterogeneous with respect to both the characteristics of their included trials and the approaches they used to assess outcome consistency. This limits our ability to pool the data to provide a summary description of discrepancies in registered and published outcomes for trials across these studies. This heterogeneity results partly from the variable time points relative to trial initiation used by authors in defining the registered outcome, and is indicative of possible bias in these results at the level of individual studies. Third, we separately analyzed the subgroup of studies that reported reviewing the history of changes within each registry record in order to identify prospectively defined outcomes; this subgroup analysis may have been incomplete because it is possible that additional studies performed this task without reporting it. Further, several of the included studies had overlapping inclusion criteria, and as a result in some cases individual studies may have been included in more than one study. Additionally, the impact of publication bias on these results is unknown. Finally, many of the included studies noted that ambiguously defined outcomes among trial registries and manuscripts made it difficult to make judgments about whether outcome differences were present.