Some lawyers are better than others and quality matters. Intuitively, this makes sense — the job of a lawyer is to persuade, which is part legal reasoning and part hard work. Since the ability to persuade directly affects litigation outcomes, participants are willing to pay highly regarded barristers upwards of $28,000 a day, and some law firm partners an hourly rate of over $1,000. In this post, we analyse data from 22,000 cases to measure the differential impact law firms and barristers have on litigation outcomes.

By applying machine learning algorithms to data mined from court judgments, we find (i) large measurable differences in expected performance and (ii) huge room for improvement in the selection of legal representation. Litigants can increase their chances of a successful litigation outcome by over 25 per cent on average, often the difference between winning and losing, by switching to better legal representation. Moreover, Chambers and The Legal 500 rankings, the prize often proudly displayed on email footers, are weakly correlated with litigation performance.

What is a useful metric to assess the expected performance of legal representation? Intuitively, having better legal representation should lead to a greater proportion of successful litigation outcomes. This points to the strategy of counting successful litigation outcomes. But measuring outcomes without accounting for the difficulty or context of the case is naive. For example, a barrister that takes the easiest cases will have a large proportion of successful outcomes and incorrectly be deemed a high performer. Secondly, litigation outcomes have a random component, outside of the control of even the best legal representation. Hence we should assess expected outcomes, instead of the random realization of outcomes.

To sidestep the randomness of observed outcomes, we focus on modeling the probability of a successful outcome for a given case. Secondly, to account for the context or difficulty of the case, we measure the change in the probability of a successful litigation outcome, from substituting a legal service provider in. The difficulty of the case is not important, but rather the impact a legal service provider has on that original probability of success. From now on, we will refer to this measure as the impact on expected performance. This substitution exercise is known as a counterfactual, as we apply the substitutions to historical data. We identify the set of better firms and barristers, for a given case, by finding those that have positive impacts on expected outcomes.

The difficulty in performing this counterfactual exercise is that many firms and barristers may have only faced a small set of other firms and barristers in previous litigation. We would like to evaluate how A would fare against B, even though we never observe a litigation matter between them. Similarly, if A is only involved in cases against the best opponents, we need to ensure we do not falsely conclude that A is below average.

Key to resolving these concerns is the fact that although many firms and barristers have not faced each other directly, they often have indirect connections to other firms and barristers via sequences of cases. Although A has not faced B, if both have faced C, then we have some information about the hypothetical A and B matchup. This is the same reasoning that allows one to compare tennis players across different eras. Although Sampras never played Djokovic, both of them played Federer. Their performances against Federer provide some guidance to how they would fare against each other. Similarly, even if Bret Walker SC only faces the best barristers, if some of these best barristers face more mediocre opposition, then we can infer how Bret Walker SC would fare against the broader set of opponents.

Figure 1: Example of Simple Graph

To quantify the connectedness of Australian firms and barristers, we use tools from graph theory. We construct a graph as follows. Each barrister and firm is a node. There exists an edge (or link) between two nodes if the two nodes have directly faced other in court. Define a path as a sequence of edges (or cases) that connects two nodes. See Figure 1 for an example of a simple graph. Ensuring that firms and barristers are comparable is equivalent to ensuring that the litigation graph is sufficiently connected, or that there are enough paths, of short length, between litigation participants. For example, the shortest path, of length 2, that connects Sampras to Djokovic consists of the matches between Sampras-Federer, and the edges between Federer-Djokovic.