By James Kwak

Ever since the financial crisis, there has been an on-again, off-again debate over the right model for financial regulation. On the one hand are those who favor simpler rules—such as a simple leverage limit based on total unweighted assets—on the grounds that they are easier to monitor and tougher to game. On the other hand are those who favor complex rules—such as the Dodd-Frank Act, which has so far generated over 8,000 pages of rules—on the grounds that the world is complicated so we need complicated rules. For the most part, this has been a shouting match over broad principles.

A friend sent me Andrew Haldane’s paper from Jackson Hole a couple of weeks ago, “The Dog and the Frisbee.” (The title refers to the ability of a dog—or a child—to catch a frisbee by following a single visual heuristic, ignoring factors such as the rotational speed of the frisbee or wind currents.) Now we have evidence.

Financial regulations have become fabulously complex in recent decades. For example, Haldane points out that since Basel II allowed banks to use internal risk management models for calculating their risk-weighted assets and capital, capital regulation is now performed by models that potentially include millions of parameters that must be estimated (p. 9). But these parameters must be estimated using relatively short historical samples—drawn from a historical period that may or may not be representative of the future.

Here we collide with another fact of statistics: when you have a limited amount of sample data, simple models have greater predictive accuracy than complex models. Haldane cites examples from a number of fields. For example, to predict the winner of a tennis or soccer match, you are better off going by the name recognition of the player or team rather than by the quantitative rankings put out by ATP or FIFA (p. 5). And when determining your asset allocation, you are better off dividing your assets evenly among all asset classes than by doing mean-variance optimization using historical data (p. 6).*

So what happens with financial regulation? Haldane analyzes various data on banks from before the financial crisis and on which banks failed (or would have failed absent government intervention) during the crisis. The conclusions? Looking at banks with over $100 billion in assets at the end of 2006, risk-based capital ratios fail to predict which would fail when the crisis hit; simple leverage ratios without weighting assets were statistically significant at the 1% level. (See Tables 3–5 for more specifications.)

The same does not hold for smaller banks, as shown by a sample of FDIC-insured banks. But that’s because of the large sample size. Haldane simulates what would happen if you estimated your models using a partial sample and then tried to predict what would happen to the rest of the sample. In that case, again, simple metrics perform better than complex models.

So what can we conclude from this? One possibility is that banks are just good at gaming the system, and the more complicated the system, the more opportunities for gaming there are. So should we vastly increase the number of regulators to keep pace with the banks and their law firms? That’s unlikely to happen in today’s environment of (self-imposed) budget constraints. More importantly, however, the real problem isn’t that risk management models can be gamed; it’s that they don’t work when the complexity of the model dwarfs the available data from which the model is estimated.

One possibility is increasing supervisory discretion. The risk there is that it would only increase the risk of (and returns to) regulatory capture. At the end, Haldane comes back to the root of the problem. The problem is complexity, and so we need to reduce complexity. One way is to tax it. Another is to insist on structural reforms to complex financial institutions. Until then, all the models in the world will do little more than give regulators the false impression that they are in control.

* Of course, if you’re predicting past performance using past data, there are superior strategies. But that’s not the point of investing, or of financial regulation.