Across 50 years of research, extensive efforts have been made to improve the effectiveness of psychotherapies for children and adolescents. Yet recent evidence shows no significant improvement in youth psychotherapy outcomes. In other words, efforts to improve the general quality of therapy models do not appear to have translated directly into improved outcomes. We used multilevel meta-analytic data from 502 randomized controlled trials to generate a bivariate copula model predicting effect size as therapy quality approaches infinity. Our results suggest that even with a therapy of perfect quality, achieved effect sizes may be modest. If therapy quality and therapy outcome share a correlation of .20 (a somewhat optimistic assumption given the evidence we review), a therapy of perfect quality would produce an effect size of Hedges’s g = 0.83. We suggest that youth psychotherapy researchers complement their efforts to improve psychotherapy quality by investigating additional strategies for improving outcomes.

Does well-designed, well-documented, psychologically principled, and carefully implemented psychotherapy lead to better outcomes than therapy of lower quality? Empirical evidence on the association between therapy quality and therapy outcome is more mixed than one might expect.

The literature reveals varying opinions on what constitutes a therapy of high quality. One view suggests that an important dimension of therapy quality is the presence of advantageous “specific factors” or “theory-specified factors” in psychotherapies (Castonguay & Grosse, 2005; Webb, DeRubeis, & Barber, 2010). These researchers emphasize the idea that the content of therapy is an important dimension of therapy quality. They stress the frequency with which certain theoretically driven approaches involving specific content, such as exposure and response prevention for OCD, outperform other psychotherapies (DeRubeis, Brotman, & Gibbons, 2005). The theory-specified factors perspective implies that randomized controlled trials comparing different types of therapy are of great importance to the scientific literature because this approach is likely to reveal the types of therapeutic content that are most effective in reducing psychopathology.

Whereas differing specific factors and treatment types have often dominated the discussion of therapy, some influential theorists and researchers have discounted the importance of these factors. Since 1936, some influential figures in the field have argued that the specific steps followed in therapy may have little impact relative to the influence of certain common factors (Messer & Wampold, 2002; Rosenzweig, 1936). One prominent version of this perspective has been labeled the “Dodo Bird” conjecture, in reference to the character in Lewis Carol’s book, Alice in Wonderland, who proclaims: “Everybody has won, and all must have prizes.” The Dodo Bird conjecture proposes that diverse types of therapies are equally effective provided that they possess certain common factors. One aspect of this perspective is the notion that across a broad range of bona fide therapies, the specific factors associated with therapy quality bear little relation to therapy outcome (Wampold et al., 1997). Importantly, the common factors approach does not discount the notion that therapy quality matters—it simply emphasizes different types of therapy quality that are independent of therapeutic content, such as adequate therapist training in facilitative interpersonal skills (Anderson, McClintock, Himawan, Song, & Patterson, 2016).

The Dodo Bird hypothesis has generated both controversy and data synthesis, with various meta-analyses and reviews cited in support of each position in the debate. Some meta-analyses failed to identify therapy type as a significant moderator of outcome (e.g., Baardseth et al., 2013; Miller, Wampold, & Verhely, 2008; Wampold et al., 1997) and have been cited as support for the Dodo Bird conjecture. Findings of other meta-analyses and some reviews indicated that the specific procedures performed in therapy matter. For instance, some of these findings identified significant between-therapy type differences in magnitude of effect size for various treated problems (e.g., Chambless & Ollendick, 2001; Hunsley & Di Guilio, 2002; Weiss & Weisz, 1995; Weisz, Weiss, Han, Granger, & Morton, 1995), and others have identified therapies for which evidence shows adverse effects (Lilienfeld, 2007). The existence of harmful therapies suggests that the dimension of therapy quality is related to therapy outcome, at least on the extreme low end of therapy quality (e.g., in which therapy is designed in opposition to psychological principles). In other relevant work, researchers have shown that type of therapy can have a marked impact when symptoms are especially severe (e.g., Lorenzo-Luaces, DeRubeis, van Straten, & Tiemens, 2017). As these examples indicate, the various syntheses of evidence have suggested that therapy content may matter in some cases but not in all. Both theoretical camps promote the idea that psychotherapy varies in its quality and its quality can be improved, but the camps differ as to how this should be done. Researchers in the specific factors camp have focused on improving therapy content, whereas those in the common factors camp have focused on maximizing factors that exist independent of therapy content, such as therapist skills in interacting with clients.

Both perspectives are relevant to the present article, in which we focus on the relation between psychotherapy quality and psychotherapy outcome. To define psychotherapy quality in a way that encompasses both perspectives, we have tried to synthesize points from both sides of the debate. The specific factors view suggests that high quality in therapy will include the use of procedures that have a basis in sound psychological principles and the accumulation of evidence from empirical studies together with training of therapists in the specific procedures involved and ensuring adherence to the specified protocols (e.g., Chambless & Ollendick, 2001). The common factors view suggests that high quality in therapy will include an array of therapist characteristics, such as skill in the interpersonal aspects of working with clients. For purposes of the present article, we include both perspectives, operationally defining quality of therapy to include both (a) the specific contents of therapy protocols and the procedures (e.g., therapist training) used to ensure faithful delivery of those contents and (b) common factors (e.g., therapist interpersonal skills) that may influence the conduct of therapy independently of specific treatment content, provided that the procedures or elements of (a) and (b) are intended to improve client outcome and do not depend on client factors.

To better understand how quality is defined and operationalized throughout this article, one can use the metaphor of a psychological scale that measures therapy quality. Our metaphorical scale for quality would include a list of items that derive from both the specific factors and common factors approach. When we refer to quality as a general principle, we refer to the metaphorical sum score of all items on the scale (or perhaps more precisely as an extracted principal component from all items). That is, we aim to represent quality as an abstract dimension comprising all relevant aspects of high-quality therapies.

How Much Does Quality Matter? As noted, researchers have argued over what constitutes a high-quality therapy. But to what degree do these different aspects of therapy quality predict outcome? We reviewed the literature on psychotherapy quality in an exploratory search. Our aim was to identify empirical articles that estimated the relationship between therapy outcome and some aspect of psychotherapy quality. Because research in this area is scarce, we broadened our review of this area to include both adult- and youth-focused therapies. To make sure we had adequately represented diverse views on this issue, we contacted prominent psychotherapy researchers and asked them to recommend studies. To identify researchers, we searched PsycINFO for the period from January 1990 to January 2018 using the terms “common factors in psychotherapy,” “empirically supported psychotherapy,” and “psychotherapy quality.” We identified authors of the identified publications who were frequently cited for work related to these topics. In addition, we identified current and past editors of journals in which psychotherapy research is frequently published. The resulting list of authors included 19 prominent researchers with diverse theoretical perspectives: David Barlow, Larry Beutler, Ronald Brown, Dianne Chambless, David Clark, Michelle Craske, Joanne Davila, Robert DeRubeis, Judy Garber, Mark Hilsenroth, Steven Hollon, Alan Kazdin, Philip Kendall, Michael Lambert, John Norcross, Francheska Pereplechikova, Dan Strunk, and Bruce Wampold. We sent an e-mail1 to each author requesting that they identify the most scientifically sound study in which the relationship between quality and outcome was assessed. Twelve of the 19 authors responded to our request. Some of the authors declined to provide a study, noting theoretical concerns with the idea of therapy quality or concerns related to unfamiliarity with more recent literature on therapy outcomes. Other authors provided more than one study. All provided studies, including results from our own initial literature review, were initially considered as part of the exploratory analysis. After reviewing the full pool of nominated studies, we excluded several studies from further analyses because of (a) failure to report effect sizes, (b) failure to include a discernable measure of therapy quality, or (c) use of therapy quality measures that depended, in full or in part, on client factors (e.g., therapeutic alliance between therapist and client). A list of excluded studies and reasons for exclusion can be found at Open Science Framework (https://osf.io/dhu7y/). The results of our exploratory search are presented in Table 1. We measured a variety of different types of therapy quality, ranging from treatment type to therapist competence to therapist facilitative interpersonal skills. Our review included both single studies of therapy quality as well as meta-analyses of evidence across many studies. Aside from comparing different types of psychotherapy, none of the studies in our search included experimental manipulations of therapy quality, indicating an important area of research that may be neglected. Table 1. Past Studies: Relationship of Indicators of Therapy Quality to Therapy Outcome View larger version A histogram of the pooled effect sizes is presented in Figure 1. A high bar in this histogram indicates that many effect sizes in the literature fell within the range of effect indicated on the x-axis. For instance, the first bar in the graph indicates that 17 effect sizes included in our review fell in the range between .00 and .02. Effect sizes are given as r2 type, which reflects the proportion of variance accounted for by therapy quality on therapy outcome (see Fritz, Morris, & Richler, 2012). In summary, most effect sizes were close to zero, indicating that higher therapy quality did not relate to better therapy outcome. Meta-analyses and more recent studies in general reported smaller effects compared with individual studies and older studies. All meta-analytic effects fell between 0 and .005. In general, these exploratory analyses showed an association between therapy quality and therapy outcome that was quite modest, at best. Download Open in new tab Download in PowerPoint

How Good Can Therapy Be? A Focus on Youth Psychotherapy After synthesizing findings of the studies listed in Table 1, we were interested in applying what could be learned from that synthesis to estimate the extent to which improving therapy quality might improve psychotherapy outcome. For that purpose, we needed a large pool of psychotherapy outcome studies. Because therapy procedures and protocols as well as required therapist skills are quite different for treatment of children and adolescents (herein youths) than for treatment of adults, we thought it best to focus on one or the other age group. Although both age groups are important, our past research on youth psychopathology and psychotherapy and the fact that we had access to data from 502 randomized controlled trials of youth psychotherapy led to our focus on therapy with young people. This work illustrates a procedure that could, of course, be applied in the future to any group, defined by age or any other factor. Using this large youth psychotherapy data set, we sought to determine the efficacy of an optimal quality therapy—that is, a therapy in which all beneficial clinician factors were maximized. In other words, we were not interested in answering the question of “How good is therapy?” but rather in answering the question of “How good could therapy be?” Answering this question is akin to an optimization problem in mathematics. First, a function must be specified that describes relevant inputs (therapy quality) and outputs (therapy outcome). The function is then analyzed to identify the point at which a maximal output is achieved on the basis of the inputs. We sought to answer this question on the basis of relevant knowledge regarding youth psychotherapy quality and outcome. To address these questions using empirical data, we first generated a bivariate distribution function, known as a copula, between therapy quality and treatment outcome drawing on an extensive meta-analysis of randomized controlled trials (RCTs) of youth psychotherapy. We then utilized our simulated distribution to predict the upper limit of effect size as therapy quality approaches infinity. In other words, we posed the question: “If we could design a youth psychotherapy of perfect quality, how effective would it be?”

Results We modeled the upper limit of treatment outcome in numerous situations, systematically varying the correlation between treatment quality and treatment outcome from 0 to 1 in steps of .01. We estimated a range for the upper limit of therapy effect size because therapy quality approached infinity by fitting linear distributions to simulations from each copula generated and approximating effect size at the 99.9th percentile of therapy quality. This analysis was done with the intent to help the reader understand how the upper limit varies as a function of the dependence between quality and therapy. As presented in Table 2, each upper limit estimate was constructed from the median of 100 samples drawn from the copula matching the given dependence level. Table 2. Estimated Upper Limit by Correlation Between Quality and Effect Size View larger version An example of a representative bivariate distribution can be seen in Figure 2. Additional examples of bivariate distributions and animated versions showing the continuous change in the distributions on the basis of different dependencies can be seen in the Supplemental Animations in the Supplemental Material. Animations are useful to show how the bivariate distribution changes dynamically as a function of the dependence and to rotate figures to see the full three-dimensional perspective. Table 2 provides insight into the upper limit of therapy effect size at various levels of dependence between therapy quality and treatment outcome. With a perfect correlation between therapy quality and outcome (r = 1.0) and maximized quality (99.9th percentile), the expected effect size (g) is 2.55—representing the highest possible value for the upper limit of therapy efficacy. If therapy quality and therapy outcome share a small to medium correlation of .2 (a somewhat optimistic assumption given the evidence we have reviewed) and therapy quality is maximized (99.9th percentile), the expected effect size is 0.83.

Action Editor

Stefan G. Hofmann served as action editor for this article. Author Contributions

P. J. Jones and J. R. Weisz jointly developed the study concept. P. J. Jones and P. Mair jointly created the methodological design and conducted analyses. S. Kuppens coded and adapted effect sizes in the meta-analytic data. P. J. Jones conducted the initial exploratory literature search, and P. J. Jones and J. R. Weisz jointly contacted researchers for further exploration of the literature. P. J. Jones wrote the initial draft of the manuscript. All of the authors approved the final manuscript for submission. ORCID iD

Payton J. Jones https://orcid.org/0000-0001-6513-8498 Declaration of Conflicting Interests

The author(s) declared that there were no conflicts of interest with respect to the authorship or the publication of this article. Supplemental Material

Additional supporting information can be found at http://journals.sagepub.com/doi/suppl/10.1177/2167702619858424 Open Practices

All data and materials have been made publicly available via Open Science Framework and can be accessed at https://osf.io/myfg7. The complete Open Practices Disclosure for this article can be found at http://journals.sagepub.com/doi/suppl/10.1177/2167702619858424. This article has received badges for Open Data and Open Materials. More information about the Open Practices badges can be found at https://www.psychologicalscience.org/publications/badges.

Notes 1.

Dear Professor ________: As part of a new study examining the relation between psychotherapy quality and psychotherapy outcome, we are contacting you and a small number of other prominent psychotherapy researchers. We ask you to please identify what you regard as the most scientifically sound study in the literature in which investigators assessed the relation between a credible measure of psychotherapy quality, on the one hand, and therapy outcome, on the other hand. If you can provide either a PDF or an exact reference for the study you identify, we will be very grateful. Thanks very much for considering our request.