Should official forecasts express uncertainty? The case of the CBO

Charles Manski

Policymakers and the media often rely on official estimates. But with policies that are so complex and often untested, these estimates are at best rough guesses – so why not be honest about it? This column calls for confidence intervals to be used in future policy debates.

Governments regularly produce official forecasts of unknown accuracy. Some forecasts become ‘conventional certitudes’ – predictions that are generally accepted as true, but are not necessarily true.

In the US, conventional certitude is exemplified by Congressional Budget Office (CBO) point predictions (or scores) of the budgetary impact of legislation. Scores are conveyed in letters that the CBO director writes to leaders of Congress. They are not accompanied by measures of uncertainty, even though legislation often proposes complex changes whose implications must be difficult to foresee.

CBO scoring of the major healthcare legislation enacted in 2010 illustrates current practice. Throughout the legislative process, Congress and the media paid close attention to the scores of alternative bills. A culminating event occurred when the CBO and the Joint Committee on Taxation (JCT) provided a preliminary score for the combined consequences of the Patient Protection and Affordable Care Act and the Reconciliation Act of 2010. CBO director Douglas Elmendorf wrote to House of Representatives Speaker Nancy Pelosi as follows: “CBO and JCT estimate that enacting both pieces of legislation … would produce a net reduction of changes in federal deficits of $138 billion over the 2010-2019 period as a result of changes in direct spending and revenue”. The letter from Elmendorf to Pelosi expressed no uncertainty and did not document the methodology generating the prediction.

Anyone seriously contemplating the many changes to federal law embodied in this legislation should recognise that the $138 billion prediction of deficit reduction can be no more than a rough estimate. Indeed, serious policy analysts recognise that scores are fragile numbers derived from numerous untenable assumptions. Considering the related matter of scoring the effects of tax changes on federal revenues, Alan Auerbach put it this way in the Journal of Economic Perspectives: “in many instances, the uncertainty is so great that one honestly could report a number either twice or half the size of the estimate actually reported” (Auerbach 1996).

Credible scoring is particularly difficult to achieve when legislation may significantly affect the behaviour of individuals and firms, by changing the incentives they face to work, hire, make purchases, and so on. Academic economists have worked long and hard to learn how specific elements of public policy affect private behaviour, but with only limited success. CBO analysts face the more difficult challenge of forecasting the effects of the many policy changes embodied in legislation, and they must do so under extreme time pressure.

In light of this, it is remarkable that CBO scores have achieved broad acceptance. In our contentious political age, CBO scores are among the few statistics that Democratic and Republican Members of Congress do not dispute. And media reports largely take them at face value.

Interval scoring

The CBO has established and maintained an admirable reputation for impartiality. Is it perhaps best to leave well enough alone and have the CBO continue to express certitude when it scores legislation, even if the certitude is only conventional rather than credible?

I understand the temptation to leave well enough alone, but I think it unwise to try to do so. I worry that someday sooner or later the social contract to take CBO scores at face value will break down. Conventional certitudes that lack foundation cannot last indefinitely. I think it better for the CBO to act pre-emptively to protect its reputation than to have some disgruntled group in Congress or the media declare that the emperor has no clothes. Hence, in a recent article in the Economic Journal, I argued that it would be better for the CBO to face up to uncertainty than to feign certitude (see Manski 2011).

How might the CBO express uncertainty? There is no uniquely correct answer, and alternatives may range from verbal descriptors to provision of probabilistic forecasts. Aiming to balance simplicity and informativeness, I suggest provision of interval forecasts of the budgetary impacts of legislation. Stripped to its essentials, computation of an interval forecast just requires that the CBO produce two scores for a bill, a low score and a high score, and report both. If the CBO must provide a point prediction for official purposes, it can continue to do so, with some convention used to locate the point within the interval forecast.

This idea is not entirely new. A version of it was briefly entertained by Alan Auerbach in the article mentioned earlier. “Presumably,” he wrote, “forecasters could offer their own subjective confidence intervals for the estimates they produce, and this extra information ought to be helpful for policymakers.” He went on to caution: “However, there is also the question of how well legislators without formal statistical training would grasp the notion of a confidence interval.”

The CBO need not describe its interval forecasts as confidence intervals in the formal sense of statistics textbooks. The main sources of uncertainty about budgetary impacts are not statistical in nature. They are rather that analysts are not sure what assumptions are realistic when they make predictions. A CBO interval forecast would be more appropriately described as the result of a sensitivity analysis, the sensitivity being to variation in the maintained assumptions.

Can Congress cope with uncertainty?

I have received disparate reactions to the idea of interval CBO scoring. Academics usually react positively, but people who have worked within the federal government tend to be sceptical. Indeed, former CBO director Douglas Holtz-Eakin told me that he expected Congress would be highly displeased if the CBO were to provide interval scores.

I have encountered a frequent perception that members of Congress, indeed policymakers more generally, are either unwilling or unable to cope with uncertainty. I am aware of no systematic research on this matter, but I have over the years heard many anecdotes. In my monograph Identification Problems in the Social Sciences, I recount a story about an economist’s attempt to convey uncertainty to President Lyndon B Johnson. As the story goes, the economist presented his forecast as a likely range of values for the quantity under discussion. Johnson is said to have replied, “Ranges are for cattle. Give me a number”.

Jerry Hausman, a long-time econometrics colleague and prominent consultant, argued for certitude to me this way at a conference over 20 years ago: “You can’t give the client a bound. The client needs a point.” I have found that many consultants share this view. They contend that policymakers are either psychologically unwilling or cognitively unable to cope with uncertainty. Hence, they argue that pragmatism dictates provision of point predictions, even though these predictions may not be credible.

British norms

Curiously, the antipathy towards measurement of government forecast uncertainty evident in Washington, DC is not as apparent in London. Since 1996, the Bank of England has regularly published probabilistic inflation forecasts presented visually as a fan chart. The fan chart provides a succinct and informative measurement of forecast uncertainty.

More recently, it has become official government procedure to require an Impact Assessment (IA) for legislation submitted to parliament. The originating government agency must state upper and lower bounds for the benefits and costs of the proposal, as well as a central estimate. The government specifically asks that a sensitivity analysis be performed, providing this guidance to agencies in its Impact Assessment Toolkit: “The ‘Summary Analysis and Evidence’ page of the IA template asks you to highlight key assumptions underpinning the analysis; sensitivities of the estimates to changes in the assumptions; and risks, and how significant they might be, to policy delivery.”

The norms for official forecasting in the UK thus differ from those in the US. I do not have a clear sense why this is the case.

References

Auerbach, A (1996), “Dynamic Revenue Estimation,” Journal of Economic Perspectives, 10:141-157.

Elmendorf, D (2010), Letter to Honorable Nancy Pelosi, Speaker, US House of Representatives, Congressional Budget Office, 18 March.

Manski C (1995), Identification Problems in the Social Sciences, Harvard University Press.

Manski, C (2011), “Policy Analysis with Incredible Certitude”, The Economic Journal, 121: F261-F289.