Today’s post is co-authored by Michael Greenstone (University of Chicago) and Catherine Wolfram (UC Berkeley) Faculty Directors of The E2e Project.

As featured in the New York Times For Government That Works, Call In the Auditors

The urgency of climate change demands solutions that work. And so, a clear-eyed assessment of how well existing policies and programs are performing is critical, no matter the results. Using rigorous evidence to inform policy design and implementation is the only way to ensure that we effectively confront climate change. The need for objective evidence is particularly important in the case of energy efficiency investments because they play a central role in virtually every single climate change mitigation plan.

Recently, we conducted a first-of-its-kind, randomized controlled trial of the nation’s largest residential energy efficiency program, the federal Weatherization Assistance Program (WAP). The trial consisted of a sample of more than 30,000 WAP-eligible households in the state of Michigan. Our research revealed that investments in residential energy efficiency upgrades among weatherized households in our study cost about double what these households will save on their energy bills (efficiency retrofits cost about $4,600 per household and estimated energy savings are only $2,400). You can read more about our results here.

These findings contradicted commonly held beliefs about the benefits of energy efficiency investments and generated a heated debate. Our previous blog post responded to some of the key concerns.

The Department of Energy (DOE) has since released 36 documents (here and here), totaling 4,446 pages, evaluating WAP. The boldly stated conclusion of the evaluation is that the benefits of the investments are four times the costs.

We have spent many hours poring over these opaque documents. Our judgment is that many of the DOE’s conclusions are based on dubious assumptions, invalid extrapolations, the invention of a new formula to measure benefits that does not produce meaningful results, and no effort to evaluate statistical significance. Using the DOE’s findings, we show below that costs exceed energy savings by a significant margin. We also document major problems with the valuation of non-energy benefits, which comprise the vast majority of estimated program benefits.

Overall, the poor quality of the DOE’s analysis fails to provide a credible basis for the conclusion that the benefits of energy efficiency investments conducted under WAP substantially outweigh the costs. This blog summarizes our assessment of the DOE’s analysis for a general audience. We provide a more technical discussion, using the important example of benefits relating to thermal stress, here.

I. Energy Efficiency Investment Costs Exceed Energy Savings by a Significant Margin.

The typical approach to assessing energy efficiency investments is to test whether the energy savings exceed upfront costs. Using standard economic assumptions, our study and the DOE study agree that costs incurred under the Weatherization Assistance Program significantly exceed energy savings. Here we focus on the ARRA period, but the basic conclusions from outside of ARRA are similar (we explain this here).

What does the DOE find? They collect energy consumption data from approximately 16,000 households. They then compare energy consumption at weatherized households before and after weatherization with energy consumption in a comparison group. Using this basic approach, the DOE finds that the average annual savings for households is $223 per year during the ARRA period. This is the primary result on energy savings.

For most, if not all, households, the decision to pursue weatherization assistance is purposeful. Households may choose to invest the effort of applying for weatherization following – or in anticipation of – a change in energy consumption or their ability to pay energy bills. In other words, there could be important differences in energy consumption patterns across groups of households receiving weatherization in different time periods. The report provides little evidence that the comparison group provides a credible estimate of the energy consumption patterns that would have been observed among weatherized households had they not received weatherization retrofits. The consequence is that we cannot know whether the measured differences in energy consumption are due to the energy efficiency investments or the pre-existing differences between weatherized households and the comparison group.

In contrast, our study is based on a randomized controlled trial (RCT) precisely because we wanted to improve the quality of evidence on the returns to energy efficiency investments. The power of RCTs is that they deliver the true effects, obviating concerns about differences in the types of households that get weatherized and the timing of those decisions. Indeed, this is precisely why society has relied on RCTs to test the efficacy of new drugs for decades and is increasingly using them to evaluate social policy.

Setting aside concerns about the estimation strategy, it is straightforward to convert the estimate of average annual energy savings reported in the evaluation, $223 per year, into a present value of energy savings over the lifetimes of the measures installed by making several assumptions. The following table describes the present value of lifetime savings under different assumptions about the discount rate and the lifespans of the measures. Notably, the table follows the standard assumption in the economics literature and assumes energy prices will remain constant, after adjustment for inflation. It also generously assumes that there is no depreciation in the savings created by the measures.

Net Present Value of Energy Savings from DOE Study of ARRA Period Time Horizon Discount Rate 3 Percent 6 Percent 10 Percent 10 years $2,003 $1,641 $1,370 16 years $3,028 $2,254 $1,745 20 years $3,646 $2,558 $1,899 Note: The table reports the net present value of energy savings (in dollars) implied by the DOE’s household-level energy savings estimates ($223) using a range of discount rates and assumed time horizons. Reductions in energy bills associated with the estimates are assumed to accrue over the life of the measure.

These “present value” energy savings can be compared against the average cost per weatherized household. The DOE reports a wide variety of costs, and it is difficult to judge which should be compared to the savings. The most straightforward is the average total expenditure per weatherized household: $6,812 during the ARRA period. The DOE reports incurring costs per household of $5,926. When comparing household benefits and costs, the DOE evaluation uses a much lower $3,745 per household, although it is unclear what cost components are omitted to arrive at this low number.

Using a 6 percent discount rate that seems a reasonable approximation of households’ borrowing costs, a 16-year time horizon (the average projected lifespan of efficiency measures in our data), and assuming no depreciation in annual energy savings (likely generous), the DOE estimate of annual energy savings implies a present value of $2,254 per household. This is less than half of the DOE expenditures per household. It also falls significantly below the lowest reported costs of $3,777.

To compute the average discounted value of energy savings, the DOE uses a longer time horizon, a lower discount rate, and assumes that energy prices will increase in real terms over time – all assumptions that make the savings look larger. These assumptions imply a higher present value of $3,190 per household. This higher value notwithstanding, WAP program costs still exceed estimated savings by a significant margin.

The conclusion that the program is remarkably cost effective cannot be justified on energy savings alone. The DOE’s claims rely heavily on the possibility of substantial non-energy benefits. We explore the credibility of these estimated benefits next.

II. Unreliable Approaches to Exploring Whether there are Non-Energy Benefits.

The DOE evaluation reports a program-wide benefit to cost ratio of 4 to 1. To arrive at this conclusion, the report points to a long list of “non-energy” benefits. They include health-related benefits such as reductions in thermal stress and asthma (valued at $6,870 per household) and improved productivity at home and at work due to improved sleep (valued at $3,142 per household), safety-related benefits such as reduced fire risk and reduced carbon monoxide levels (valued at $985 per household), and other benefits such as reduced reliance on food and energy assistance programs (valued at $1,641 per household). The rationale for relating energy efficiency investments to some of these outcomes (e.g., improved productivity due to better sleep) is not well articulated, but nevertheless we examined the basis for the claims.

A close look at these claimed benefits uncovers significant problems with the methods used and assumptions invoked. We highlight three general concerns here.

Non-energy outcomes are not directly measured. And the path from imputed impacts to benefits estimates is generally dubious.

In contrast to energy consumption, which is measured directly using household energy bills, most of the non-energy benefit estimates are based on survey responses from program participants. Connecting the dots between survey responses and non-energy benefits involves many steps. The resulting benefit values are extraordinarily sensitive to many key assumptions, several of which are not well explained or substantiated.

The significant benefits attributed to reduced hospitalization rates after weatherization provide one example. DOE surveyors asked a select set of households the following question: “In the past 12 months, has anyone in the household needed medical attention because your home was too cold (hot)?” Using a formula they invented that has never appeared in any textbook (more on this below), the DOE deduced from the survey responses that weatherization assistance reduced the share of households needing medical attention by 1.4 percent.

The path from the response to this survey question to monetized benefits is long and winding. The nearly $5,000 per household of mortality reduction benefits are particularly illuminating. Since no data on mortality was collected, the path to claimed reductions in death begins with the above question about medical attention. Of course, not all instances of medical attention lead to death, so the DOE analysis assumes that some proportion of these instances lead to hospitalizations, doctor visits, or emergency room visits. The next step in this circuitous path is to assume that some fraction of the assumed hospitalizations lead to death. Consequently, responses to a survey question about “medical attention” turn into $5,000 worth of benefits per weatherized household.

This approach to calculating benefits is applied throughout the analysis of non-energy benefits. For example, responses to questions about sleepless nights are converted into productivity benefits that exceed $3,000 per household. The general point is that the failure to directly measure the outcomes of interest leads down a long and winding road that seems far removed from solid footing.

There are three other noteworthy points about the imputation of health-related benefits

It is natural to assume that the starting point of benefits associated with reduced thermal stress would be a substantial change in indoor temperatures, making it warmer in the winter and cooler in the summer.In the only report where indoor temperature is directly measured, DOE finds that average indoor temperature increased by 0.3 F in weatherized homes as compared to a control group. This very small change calls into question the basis for the claimed health effects due to reductions in thermal stress.

Notably, in cases where DOE analysts were able to directly measure outcomes with a documented connection to health, these data were not systematically used in the benefits valuation exercise. For example, DOE researchers conducted a field study of indoor air quality parameters in weatherized homes, including carbon monoxide (CO), radon, formaldehyde, and temperature. This study found no significant changes in carbon monoxide following weatherization. Yet, the claimed benefits include reductions in CO poisoning.

In contrast, weatherization was found to increase radon and formaldehyde levels. In the accounting of costs and benefits, however, no effort was made to quantify the potential health costs from the increase in these potential health risks. Thus, at least in the case of the direct indicators of air quality, the DOE study assigns benefits in cases when direct measures do not change and fails to evaluate costs when the indicators worsened. This selective accounting is a cause for concern.

The invention of a new statistical technique to estimate benefits.

A very standard approach to estimating the impacts of an intervention (such as weatherization assistance) on an outcome, absent a randomized controlled trial, involves taking the difference in the pre-post change in outcomes in the treated group and then subtracting the pre-post difference in a comparison group. This is the basic approach that the DOE used to calculate energy savings summarized above. If outcomes observed in the comparison group provide a good proxy of what would have been observed in the treated group absent the treatment, this approach can provide a credible estimate of the effect of a program.

To estimate impacts on key non-energy outcomes (such as health and productivity), the DOE introduces a very non-standard and non-intuitive approach. Specifically, DOE analysts compute the average of the observed pre-to-post change in outcomes in the treated group and the observed difference in outcomes between the control group and the treatment group in the pre-period. This approach takes an average of two differences, rather than subtracting one from the other. In doing so, it undermines the entire rationale for the difference-in-difference approach (namely to control for the effect of potentially confounding changes over time that are not caused by the weatherization).

To the best of our knowledge, this approach has never been used in any textbook or research paper previously. Additionally, DOE researchers provide no rationale. While the DOE’s analysis of energy savings requires some non-trivial assumptions that are not required for randomized controlled trials, it is at least a recognizable approach to estimating the impacts of the program. In contrast, the approach to estimating non-energy benefits is unrecognizable, and we believe the resulting estimates have no meaningful interpretation.

No attempt to separate a signal from the noise.

People’s lives change from day to day for many reasons. For example, the number of work days missed, or prescriptions filled, can vary over time for a variety of reasons most of which cannot be measured. The standard practice is to test whether a program’s impacts are statistically significant or just due to random chance. The evaluation does not follow this important convention. Indeed, the little information that is provided in footnotes about the variability of outcomes in treatment and comparison groups suggests that many of the key differences documented could well be due to random chance versus a causal impact of weatherization.

In summary, non-energy benefits comprise a very significant share of the estimated benefits of the energy efficiency investments that were made through the WAP program. The process of assigning dollar values to non-energy costs and benefits is inevitably complicated. Estimated values can be highly sensitive to underlying assumptions and measurement choices.

We find the DOE’s evaluation of non-energy benefits is flawed in several respects. While it is certainly possible that the people who received weatherization improvements benefitted in other ways, our judgment is that the DOE analysis fails to provide evidence that can credibly confirm or contradict this possibility.

Conclusion

The DOE has gone to great lengths to collect data on what happened to households after they received weatherization assistance. For this, they should be commended. However, the data were not analyzed in transparent and standard ways; this undermines the credibility of the results.

A seriousness of purpose in achieving policy goals requires finding out what works. A commitment to independent evaluation is the critical first step, and that requires high-level political support. The Obama Administration has supported evidence-based policy making in other realms; there are great opportunities to extend these efforts.