Balkinization: Should we Really Have Relied on Cost-Benefit Analysis? A Response to Ian Ayres, Yair Listokin, Robert Schonberger, and Zachary Shelley

Should we Really Have Relied on Cost-Benefit Analysis? A Response to Ian Ayres, Yair Listokin, Robert Schonberger, and Zachary Shelley

Guest Blogger

Nik Guggenberger

In a post on this blog, Ian Ayres, Yair Listokin, Robert Schonberger, and Zachary Shelley posed the question, “Should We Really Have Shut Down a Week Earlier?” As the title suggests, the authors do not think so. They argue that it was right to let 36.000 people die instead of sacrificing another $ 40 billion in economic output. While the authors offer a valuable account of some of the trade-offs that that public health measures in a pandemic entail, their calculation of costs and benefits proves incomplete. Structurally, it demonstrates some of the limitations of cost-benefit analysis for policy making.

The authors pick up on a much-cited study by Sen Pei, Sasikiran Kandula, and Jeffrey Shaman, which suggests that 36,000 lives could have been saved if the nation had gone on lockdown one week earlier. Ayres, Listokin, Schonberger, and Shelley then criticize the researchers’ conclusion that their “findings underscore the importance of early intervention and aggressive response in controlling the COVID-19 pandemic” and the media narrative based on the study, namely the suggestion that “our failure to shut down earlier was an unmitigated disaster.” Instead, they argue that “after-the-fact cost-benefit analysis rejects an earlier shutdown as too costly,” or at least proves that the decision was a close call.

Cost-benefit analysis is a widely-used policy tool that, as its name suggests, analyses the costs and benefits of policy decisions. The method emphasizes that we constantly face trade-offs. While a lower speed limit might reduce traffic casualties, it might also prolong our commutes to work. Or, as the authors point out, if we had shut down the economy earlier, we would have saved many lives, but we would also have reduced economic output and harmed the very people we aimed to protect. Cost-benefit analysis attempts to quantify the upsides and the downsides of a given policy. It asks us to pursue a policy whose benefits outweigh its costs.

The basic theoretical approach is this: Life is complicated. There are always trade-offs. We should quantify all trade-offs and subtract the costs from the benefits of a policy. If the sum is positive, the policy is a good one. If the sum is negative, it is a bad one. In practice, though, measuring and balancing is not as easy as it seems. In fact, it may blur our vision more than it enlightens us.

Take the issue of trade-offs. The authors point out that if we had gone on lockdown one week earlier, we would have saved 36,000 lives, but we also would have forgone more economic output. The authors quantify the benefits of an earlier lockdown due to the lives saved at $34 billion (“$125,000 per year of life in good health”) and the costs due to reduced GDP at $40 billion. The costs are greater than the benefits, so we should not have shut down the economy a week earlier to save 36,000 lives.

But matters are much more complicated than this calculation suggests. To their credit, the authors admit that their “estimates of both the costs and the benefits are not precise” and that “slightly different assumptions” might have tipped the scale in the other direction. But the real shortcoming of the calculation lies elsewhere. It concerns neglected consequences, false trade-offs between saving lives and saving the economy, and the failure to discuss mitigating policy levers. The authors’ focus is a comparison of “mortality benefits of a March 8 shutdown” with a reduction in GDP. But the analysis of trade-offs should be far broader than this.

First, consider the benefits-side of an earlier shutdown: Should we only consider lives saved? Or should we also consider health care costs and suffering caused by non-lethal, but severe COVID-19 cases? What about the physical and mental toll on health care workers? Should we ignore the distributive impacts of an earlier shutdown, or should those consequences not be considered as benefits? What about the consequences for social justice and racial disparities? (Although the authors note these issues, they do not include them in their calculation). Then there are the political risks. Would an earlier shutdown have been a better hedge against the dangers of a slide toward authoritarianism or against the risks that this country will be unable to hold a fair election in November? Should we care about the positive environmental impact of the reduction in traffic?

Next, consider the costs-side of an earlier shutdown: Is the economic output lost forever, or is it just postponed? (If the latter, we would have to factor in the economic costs of postponement.) Indeed, the economy unexpectedly added 2.5 million jobs in May and the S&P 500 has made up for its losses incurred in March, suggesting that economic recovery might already be under way. Would an earlier shutdown not also have meant that we could (have) re-open(ed) much earlier? (Potentially more than one week earlier.) Evidence from the Spanish Flu in 1918 suggests that regions that reacted quickly and aggressively fared better economically. Could a more rapid lockdown have been less strict and still saved (close to) 36,000 lives while imposing significantly lower economic costs?

How easily can we compare losses in societal revenue (GDP) with losses in what the authors call societal wealth—that is, human lives? Would people have accepted lockdowns before the death toll had risen to March 15th-levels? Could we have combined an earlier shutdown with policies that would have mitigated its economic impact?

On both the cost and the benefit sides of the balance, there are important questions about what impact an earlier shutdown order might actually have had. The authors write that individual precautionary measures did not impact the economy much before March 15. But location data—perhaps the most immediate feedback available—suggests otherwise. In fact, people hunkered down days before the official lockdown orders were put in place. This last phenomenon is especially significant when basing the effects of a policy measure on a period as short as one week. Anecdotal evidence about the re-opening of society implies the same: When visiting a local business last Saturday after the lockdown was eased in Boston on May 25, I asked an employee whether it had been busy since. He replied: “No, people are scared.” Empirical and anecdotal evidence points to the same conclusion: a significant portion of economic losses would have occurred with or without an earlier official shutdown. The study by Sen Pei, Sasikiran Kandula, and Jeffrey Shaman estimating the lives saved by an earlier shutdown includes actual mobility data.

All this is to show that life is more complicated than the relatively simple cost-benefit balancing the authors put forward—especially given their inevitable hindsight bias. Cost-benefit enthusiasts could point out that we inevitably always ignore many factors in decision-making processes or, when analyzing past policy decisions, that policymakers deliberately focused only on specific implications. Yet such replies undermine the alleged advantages of cost-benefit analysis. If we accept that people will pick and choose what costs and benefits to consider, it threatens to turn the method into little more than an ideological tool detached from reality.

We can, of course, try to enrich the calculation of the blog post’s authors by including more data points. Yet this would ignore the larger, systemic flaws of cost-benefit analysis as a policy tool, flaws that the authors themselves are no doubt familiar with.

As commonly practiced, cost-benefit analysis inevitably favors certain implications over others, simply some are harder to measure than others. For that reason, the approach tends to exaggerate static considerations over dynamic effects. The method all too often glosses over distributive consequences and ignores the independent value of “non-economic” factors by trying to squeeze them into an economic corset. It confuses price theory with democratic values: the hypothetical underlying market ignores that preferences are distorted by wealth effects and many other “artificial” (that is, real-life) constraints. Policy choices based on the financial ability to act in accordance with (revealed) preferences will inevitably perpetuate systemic inequalities. And the underlying price theory also leads to a disregard for intergenerational justice. Each of these considerations limits the value of cost-benefit analysis as a policy tool.

It is important to recognize that criticizing cost-benefit analysis as it is currently performed does not mean rejecting its most important insights: we should not ignore trade-offs in policy-making, and we should not refrain from quantification where we can measure impacts without distorting the incentives of the actors. The problem is not in these theoretical assertions. Rather, it is that, at least in its current form, cost-benefit analysis has become the voodoo economics of policy-making. Only a much humbler version of cost-benefit analysis, one that acknowledges the method’s limitations , can provide any measure of the benefits it promises.

Nikolas Guggenberger is the Executive Director of the Information Society Project at Yale Law School. You can reach him by e-mail at nikolas.guggenberger at yale.edu

Posted 2:31 PM by Guest Blogger [link]