Balkinization: So Long and Thanks for All the F-Tests

So Long and Thanks for All the F-Tests

Ian Ayres

Crosspost from Freakonomics:

I’ve been reading a truly excellent book by Joshua Angrist and Jorn-Steffen Pischke called Mostly Harmless Econometrics: An Empiricist’s Companion. It’s not written for a general audience, but if you pulled an A- or better on a college-level econometrics course (and if you love Freakonomics), then this is the book for you. It should be required reading for anyone who is trying to write an applied dissertation. It is the rare book that captures the feeling of how to go about trying to attack an empirical question; and it does this by working through two or three dozen of the neatest empirical papers of the last decade (often coauthored by Angrist). It is also peppered with references to Douglas Adams’s writing — so what’s not to like?

Here’s a fine example, in plain English, explaining how econometricians think about what they are doing:

[Something that distinguishes] the discipline of econometrics from the older sister field of statistics … is a lack of shyness about causality. Causal inference has always been the name of the game in applied econometrics. Statistician Paul Holland (1986) cautions that there can be “no causation without manipulation,” a maxim that would seem to rule out causal inference from nonexperimental data. Less thoughtful observers fall back on the truism that “correlation is not causality.” Like most people who work with data for a living, we believe that correlation can sometimes provide pretty good evidence of a causal relation, even when the variable of interest has not been manipulated by a researcher or experimenter. (p. 133)

The book backs up this assertion by teaching the reader to think carefully about what assumptions about the counter-factual are necessary to make a causal inference. I was thinking about the book a couple of weeks ago when reading a New York Times article discussing the college and law-school years of Supreme Court nominee Sonia Sotomayor. The article in the second paragraph claims that Judge Sotomayor “benefited from affirmative action policies.” To me, this is pretty clearly a causal claim and this claim is not well supported by the subsequent evidence in the article.

At least one relevant counterfactual question to ask is “What would have happened to Judge Sotomayor in applying to college and law school in a world without affirmative action?” We are told that Ms. Sotomayor was an honors student in high school and that she graduated near the top of her class in college. James A. Thomas, a former dean of admissions, concluded that “Ms. Sotomayor’s background had little role in her acceptance to [Yale Law] school.” This is hardly strong evidence for claiming that she was a beneficiary of affirmative action. The article shows that it is not just econometricians who can mistake correlation for causation. It is a mistake that a reader of Angrist and Pischke is less likely to make.

Posted 2:36 PM by Ian Ayres [link]