Skepticism About Leiter's Citation Rankings

Brian Tamanaha

Like Mary Dudziak, I’m skeptical about Brian Leiter’s latest citation rankings of law professors. In response to Dudziak’s concerns about the study, Leiter claims that “The study is a ‘true measure’ of what it purports to measure, namely, impact in legal scholarship.”

It is more accurate to say that citation counts measure citations. Whether citations reflect impact is an altogether different question.

My objection to use of citations as a proxy for “impact” is not the claim that articles and books may have an influence without being cited in law review articles, although this is clearly the case. [I have read and learned from Isaiah Berlin, for example, but have never cited him].

Rather, the problem has to do with the bizarre citation practices that have developed in U.S. law reviews. Law reviews typically require that almost every assertion be backed up by a reference; articles often have in excess of 400 footnotes, nearly one for every sentence [invited and symposium pieces escape these constraints].

As a result, law professors are required to produce reams of citations, even for commonplace assertions, a task they sometimes push off on research assistants. Over time, stock or standard citations develop, which are cited again and again. An easy way to come up with a citation is to plumb (or loot) the footnotes of earlier articles on the subject. A lot of parasitic opportunism of this kind takes place because it is an efficient way to come up with the required footnote. [Concrete evidence of this practice: an article I wrote a dozen years ago was erroneously identified as co-authored in an early article that cited it, and to this day it is still occasionally cited as co-authored, which suggests that the people who cite the article are not actually reading it but are merely "borrowing" the (erroneous) cite from earlier articles.]

Owing to this practice (common?), the fact that a book or article is cited does not necessarily indicate that it was read by the law professor who cited it. Even if the professor actually reads it, moreover, the citation does not mean the article or book cited had any impact on the professor, particularly when the citation is produced after the passage was written. Again, many sources are cited solely because a citation is required by law reviews.

A more refined measure of impact or influence would count only the times when a source is actually discussed in the article in some fashion, even minimally. While there is much reason to doubt that a stock cite (or single reference in a string cite) is a reliable measure of impact, there is no question that actually engaging with the article or book constitutes impact.

Even if this problem is corrected, there are other serious problems with Leiter’s citation study as a measure of impact on legal scholarship.

Consider, for example, Leiter’s ranking of Critical Theorists. Roberto Unger is ranked 20th, with 480 citations. Setting aside what one might think of the merits of critical theory, it is absurd to suggest that the “true measure” of Unger’s impact in this field places him behind all the others cited. His Knowledge and Politics and Law in Modern Society influenced a generation of critical theorists (and others), although these works might not be cited very often today. This example alone demonstrates that the citation study is deeply flawed as a measure of impact.

Take a look at the “Law & Philosophy” ranking. A case can be made that Duncan Kennedy (1290 citations) and Roberto Unger, both relegated (or banished?) by Leiter to the Critical Theorists list, should also have been included on this list (both placing in the top ten, with Kennedy second). Leiter will no doubt assert that they do not engage in “legal philosophy” proper, which is a plausible claim, though by no means uncontroversial (Nussbaum and Waldron, on the list, also do much work that does not fit within a narrow definition of "legal philosophy"). Even conceding this, one might ask why such a narrowly defined category was utilized that excludes such important contemporary legal theorists.

Another general problem with the ranking is that many people are cited for work in other fields: Raz for moral theory; Waldron for political theory; Leiter for his rankings; and so forth. This is true for many professors, not just those in legal philosophy. Leiter does not correct for this, which undermines the accuracy of the rankings (relative position and who makes the cut).

Leiter, to his credit, admits that there are flaws in his citation ranking, although he nonetheless believes that it offers a valid way to measure scholarly impact.

However, it may well be that there is no way to truly measure scholarly impact, which is impossible to quantify. If so, even a measure that is arguably “better” than its competitors is worse than no measure at all because it falsely suggests that it is measuring something which cannot be measured.

Our culture suffers from a ferocious ranking fetish. Leiter’s citation study feeds the beast, when we should instead be starving it.
Posted 3:33 PM by Brian Tamanaha [link]