I

Analysis of 28,000 proceedings in California suggests that appellate specialists do not perform much better.

This is the second part of a two-part series on appellate specialists in litigation.n the first part of this series , we explained the differences between appeals and trials in the United States that help to explain the existence of appellate specialist lawyers in the United States and their absence in other jurisdictions. To help determine whether it makes sense for litigants and trial lawyers to brief appellate lawyers, we analyzed the performance of law firms in over 28,000 civil proceedings in the California Courts of Appeal from the past five years. As we outline below, surprisingly, appellate specialists are only marginally better than non-specialists at obtaining reversals of trial decisions.To assess the relative performance of law firms, we compared appellate specialists against non-specialists using a statistical model that infers the impact that a law firm office has on the probability of obtaining a reversal. The model is trained on data from civil appeals in the California Courts of Appeal and accounts for the nature of the appeal (whether an as-of-right appeals or a writ petition, grounds of appeal, area of law, and the panel and the court involved).Accounting for the nature of the appeal is important as it allowed us to make apples-to-apples comparisons — that is, when assessing the performance of a firm in an appeal — we were able to compare a firm’s performance in each appeal against the expected outcome in an appeal with the same characteristics. It also ensures that a firm that focuses on writ petitions (which are less likely to succeed given their discretionary nature) is not incorrectly inferred to have a negative effect on obtaining a reversal in an appeal.One caveat is that unmeasured variables, such as the difficulty of cases a firm has tried, may affect the estimates. So if a firm does only handle fundamentally “difficult” appeals (not due to any of the measured variables), it will incorrectly appear to have a negative impact. This impact is muted when it comes to appeals because appeals are rather similar to each other (i.e., they’re much more similar to each other than trials are).As previously demonstrated, appeals are narrower in scope, do not require decision making by juries or allow for the adducing of evidence (usually), and are much shorter in duration. The standards of review required to be met by appellants also makes it more likely that appeals are of a certain standard and fall within a similar range of meritoriousness. All of this limits the effect of unmeasured variables, such as difficulty, on our analysis.

Related: How to Create and Execute a Strategy-driven Budget







Figure 1: Posterior distributions of firm alpha parameters



Our model works by assigning each firm a value (call it “alpha”), which governs the impact the firm will have on the chances of a reversal of a typical appeal. We start with the prior assumption that each firm has the same impact as others, while allowing for a significant degree of uncertainty. After observing the outcome of an appeal and the result achieved by the firms involved (e.g., reversal or dismissal, grant or denial of petition), we update the assumptions for the firms. We do this for every firm that has been involved in an appeal in the California Courts of Appeal in the last five years.Because we measured the factors making up the nature of appeal (e.g., areas of law, court, amongst others), we were able to separate the effect of these factors from the estimated impacts for each firm, and also able to quantify the factors underlying the success of an appeal. For example, we estimate that the success rate of a writ petition is less than half that of an ordinary appeal, all else being equal. By estimating the effect on success rates due to the proceeding being a writ petition, we neutralized the bias that would otherwise be generated by firms that appear in an above-average or below-average proportion of writ petition proceedings.Figure 1 shows the model’s estimate of the probability distributions of the estimated impact of roughly 11,000 law firms (each office of a multi-office firm is treated as a separate firm) on an appellate proceeding. The shape of the curves indicate certainty, with more peaked curves indicating greater certainty about the firm’s impact. Positive impacts are those to the right of 0 (y-axis).To compare firms, it is preferable to calculate the probability that one firm will perform better than another, rather than comparing each firm’s mean (or average) estimated impact, particularly if the distributions are shaped differently. This is because the mean does not account for the uncertainty in the estimates. Graphically, this amounts to calculating the overlap in areas underneath each firm’s distribution curve. Intuitively, firms with similar performance will have a greater overlap.Before looking at firms generally, we’ll focus on the three firms ranked in the Tier One category of Chambers and Partners Litigation - California Appellate section (we leave these firms unnamed). These firms are highlighted in Figure 1. As expected from firms ranked highly in appellate law, all three firms have positive impacts on reversal rates. The model suggests, however, that their performance varies widely.Appellate Firm A dominates both Appellate Firm B and Appellate Firm C, with appeals undertaken by Appellate Firm A 14 and seven percentage points (using the mean of the estimates because the distributions are similarly shaped) more likely to lead to reversal. Comparing Appellate Firm C to Appellate Firm B is more difficult since there is greater uncertainty regarding Appellate Firm C (indicated by the wider shape of the curve). Relying on the method described in the previous paragraph, the probability that Appellate Firm C’s impact is greater than Appellate Firm B’s is 0.78 (or as most people will think about it, 78 percent).

Related: How to Make the Case for a Smaller Law Firm to Your Board







Figure 2: Distribution of alpha parameters for appellate specialist and non-specialist law firms



Talented litigators have universal skills

For this example, it is possible though unlikely that the differences in the estimates are due to differences in the underlying difficulty of appeals at each of the firms. All three firms are prestigious appellate firms based in Southern California, likely having a similar portfolio of appeals. We also note that the model asserts probabilities and allows for uncertainty.Looking more broadly, we compared the estimated impact for appellate specialist firms (defined as those that have appeared in at least 10 proceedings and for which appellate proceedings constitute at least one-fourth of their total appearances) against the estimates for the balance of firms.Figure 2 shows the distribution of estimates for appellate specialist firms against non-specialists. Although it is hard to see, the distribution for appellate law firms is marginally shifted to the right when compared to non-specialist firms. More precisely, there is a 63 percent chance that the appellate firm will perform better — that is, in any given circumstance, a randomly chosen appellate firm is only likely to marginally outperform the non-specialist. That chance is slightly higher than a coin toss.What’s more surprising is that there are also poorly performing appellate firms. “Poorly performing” in this context means that the firms have a negative impact on the outcome of the appellate proceeding. It’s possible to infer the existence of these “poorly performing” firms as Figure 1 shows appellate firms with a positive impact exist and Figure 2 indicates the existence of firms that shifted the otherwise positive distribution towards zero. These poorly performing appellate firms actually perform worse in appeals than the average non-specialist firm. We note, however, that the thinner distribution curve for appellate firms suggests that appellate firms are less likely to perform poorly.How do we reconcile this negligible boost in impact with the differences between trials and appeals, and the different skill sets required for the proceedings set out above? It suggests that there’s greater transferability in the skills of trial lawyers than appellate lawyers would be ready to admit.

Related: 3 Ways to Motivate Your Litigation and Legal Teams



Notes for corporate counsel

Sidebar references

Written submissions and briefs are less important in trials, but trial lawyers often submit written outlines of arguments on particular legal issues. They also engage in limited oral argument. Good skills of persuasion also seem to be of general applicability, and be effective whether the target is a jury or a panel of judges.Finally, critical thinking skills aren’t exclusive to good appellate lawyers.To be fair to appellate specialists, it’s also worth noting that the data does show that a randomly selected appellate specialist firm is expected to perform better (albeit only slightly) than a randomly selected non-specialist. This suggests that there is some value to the specialized skills of appellate lawyers.For corporate counsel, the preceding analysis suggests some guiding principles. If you’re interested in undertaking a data driven approach, you can do so without specific analysis drawn from the statistical model.Where appellate specialist lawyers exist (i.e., in the United States), first consider whether the lawyers that were retained at trial have experience on appeals. Appeals are very different from trials. To give yourself the best chance of success, ensure that your lawyers have experience handling appeals. Further, because experience allows you to be more certain of a lawyer’s track record, a large number of appeals with an above average track record should be assessed as better than a tiny number of appeals with a stellar track record. When engaging an appellate firm, ask prospective firms about the number of appeals recently undertaken in the same area of law.Secondly, avoid making a simplistic comparison of reversal rates as the baseline reversal rate differs quite widely according to the nature of appeal. Instead, ensure you are making apples-to-apples comparisons.In countries where appellate specialist lawyers aren’t prevalent, selecting a new set of lawyers makes less sense because there’s less of a difference between the trial and the upcoming appeal. There’s therefore, less of an obvious upside to switching teams, particularly when, as Albert Ounapuu states, “The knowledge of the lawyers that conducted the case at first instance about how the case was won or lost will be essential.”If you’re considering another set of lawyers, and appellate specialists don’t exist, think carefully about how difficult it will be to transfer knowledge to the new team.

Worldwide: Chambers and Partners, Legal500

United States: Lex Machina (Federal Court), Litimetrics (State Court)

Australia: Doyles Guide

European Union: JUVE Handbook (Germany)

Canada: Canadian Lawyer Magazine