Doubts about the New "Scholarly Impact" Ranking

Brian Tamanaha

A new law school ranking is out which purports to objectively measure the “scholarly impact” of the “Top 70” law faculties. The ranking has generated a great deal of interest and commentary among legal academics. Reflecting this interest, their paper has been downloaded over a thousand times in little more than a week (earning a high rank for number of SSRN downloads within two weeks of posting). While the authors concede that their study, which extends Brian Leiter’s ranking of the top 25 law faculties, has limitations, they assert that it is superior to the US News ranking.

The US news ranking might well be garbage, as many law professors complain, but it is not a persuasive reason in favor of the new ranking system to assert that it is superior to garbage. If we are to take it seriously, it must constitute a valid way of measuring what it purports to measure: “scholarly impact.” I will raise a few doubts about the validity of this new ranking system.

A core theme of their paper is that several law faculties are treated unfairly by the US News Academic Reputation rating. “Based on Scholarly Impact Scores,” the authors write, “several law faculties appear to be significantly under-valued law schools.” For example:

Presumably due to its recent entry on the scene, the University of St. Thomas (Minnesota) is the most dramatically under-valued law school among the top 40 in scholarly impact. The University of St. Thomas enters into the First Tier at #38 in the Scholarly Impact Ranking, while being ranked by U.S. News in the Third Tier.

The authors of the study, it bears noting, are professors at St. Thomas. I sympathize with their frustration. St. Thomas has an excellent faculty (I have visited the school, and two dear friends of mine, leading scholars in their respective fields, are on their faculty.).

What the authors fail to mention, however, is the reverse implication of their contention. For every law faculty that is under-valued, there must be a law-faculty that is over-valued. This raises uncomfortable questions. Are the authors claiming that their faculty has a greater “scholarly impact” than, say, the law faculties at Iowa, Alabama, San Diego, William & Mary, Fordham, Florida, Wake Forest, Wisconsin, Boston College, Tulane, etc.? (just a few of the 150 or so law faculties that score below St. Thomas).

A more dramatic example will help illustrate my point. Cal-Irvine, a brand new law school, is ranked 9th by the “scholarly impact” study. Would the authors say that the Cal-Irvine faculty (impressive as it is) has a greater “scholarly impact” than the law faculties at Cornell, Duke, Michigan, Pennsylvania, UCLA, Virginia, Vanderbilt, Texas, Georgetown, Minnesota, Illinois, etc.? (all ranked below Cal-Irvine).

These claims strike me as patently implausible. And I suspect many law professors would have the same negative reaction. This contrary intuition suggests that perhaps there is something wrong with how the authors have constructed the ranking or with what they claim the ranking represents (of course, it is also possible that my contrary intuition is wrong). If the authors are not in fact making these claims, then what are they claiming?

This is not just about where St. Thomas and Cal-Irvine stand. All of the law faculties mentioned above are talented and prodigious. The proposition that each can be precisely slotted relative to the others by “scholarly impact” calls out for skepticism.

Not surprisingly, the closer one looks at the study the less sound it appears. To produce their ranking (following Leiter’s methodology), the authors count the number of times tenured law professors at each school are cited in the Westlaw law journal data base (more accurately: they count the number of times a professor’s name is mentioned, including acknowledgments and other non-substantive references). They multiply the mean citation count of the tenured faculty by 2 and add this to the median citation count.

If we are truly measuring the “scholarly impact” of a law faculty—and if we assume that citations are a good proxy for scholarly impact (more on this later)—then it seems that the most straightforward way to compile the ranking is simply to add up the total number of citations for all professors on each faculty.

The smaller schools will protest that this would put them at a disadvantage, but why should that be a reason to adjust how we count? If the ranking purports to measure the scholarly impact of each law faculty, size matters. Cal-Irvine has a small group of frequently cited professors (boosted by the presence of the most highly cited person in legal academia, Erwin Chermerinsky), but it can hardly be maintained that their “impact” exceeds that of larger faculties with a greater total number of citations (think of Texas or Georgetown). It might even be said that the study design hands an unwarranted advantage to small faculties that possess a couple of highly cited scholars (because the mean—which counts double—is more subject to influence on a small faculty).

Another counter-intuitive oddity lies in the details of the study.

The authors presumably would agree that there is no real difference in “scholarly impact” between schools ranked right next to each other. It seems reasonable to assume, for example, that the faculties ranked 27th and 28th are roughly equal, especially since their citation scores are not far apart. A real difference in “scholarly impact” exists only between schools that have markedly different citation scores—like the 200 points that separates the school ranked 14th (Penn) from the school ranked 23rd (Emory). Right?

Not so fast. It turns out that Yale (#1) scores 200 points higher than Harvard (#2). Is it plausible to say that Yale exceeds Harvard in “scholarly impact” by a significant margin? Consider further that Yale’s weighted score is almost 600 points above Columbia’s (#6). Yale’s score exceeds Columbia’s by more than a third. For perspective, note that the next school on the list, Berkeley (#7), is 600 points above Wisconsin (#55). In terms of citation score, then, Yale is superior to Columbia in “scholarly impact” to the same extent that Berkeley is superior to Wisconsin.

It should be evident that there is something problematic with measuring “scholarly impact” in this way. I doubt that even the folks at Yale believe they are so superior to Harvard and Columbia (and Chicago, Stanford, NYU, etc.). These fabulous faculties are interchangeable (literally, professors at these schools regularly move from one to the other, and back). Indeed, in this respect the US News Academic Reputation rating appears to be more accurate than the scholarly impact ranking because the former has Yale and Harvard in a tie, followed closely by Chicago, Stanford and Columbia in a tie.

Another problem with the study becomes apparent when we understand why Yale fares so well. Substantially more articles are published in law journals on constitutional law than any other subject. Consequently, constitutional law scholars are cited far more often than other scholars. (When ranked by citation count, for example, the twentieth ranked constitutional law scholar has more citations than the first or second ranked scholar in almost all other fields.) Yale scores so highly in large part owing to its extraordinary cadre of constitutional law scholars—with five faculty members placing among the top ten in citation counts in this category. Yale has a terrific faculty across the board, of course, with leading scholars in every field. But so does its peer schools. What sets Yale apart in citation counts is its particular strength in constitutional law, combined with the fact that constitutional law is the most written on topic.

Is it sensible to construct a “scholarly impact” score that is skewed by academic fashions? The infatuation of law review editors with constitutional law is not quite a fad—it has been dominant for some time—but I have never seen a persuasive argument that constitutional law scholarship is the most important or influential legal field. Perhaps a more accurate ranking of “scholarly impact” would equalize the weight of citation counts across legal fields.

I don’t have an answer to this question. I raise it to show, once again, that this new ranking system is problematic in fundamental ways.

When raising these doubts, I have deliberately stayed away from a pivotal underlying question: Is a citation a reliable indication of “scholarly impact”? The legal academy follows a notoriously bizarre citation system: virtually every assertion in an article must be supported by a citation (example: “The rule of law is an important notion around the world today.” See Tamanaha, On the Rule of Law). This requirement inflates citation counts; law professors often rely upon research assistants and law review editors to produce the obligatory citations; often the students produce these citations by raiding the footnotes of other articles that previously made the same point; a source cited once can thereby become a source cited many times (dirty secret: one cannot assume that a source cited was actually read by the professor who cites it.). This underside of citation practices—which throws doubt on the proposition that citation counts are a reliable measure of scholarly impact—is well known.

Some of my concern would be allayed if the authors renamed the study with a more accurate title: “Ranking of Law Faculties by Mean (x 2) plus Median Citation Counts.” To claim anything more than that is unsupported.

The authors might respond that if I don’t like their study I should devise a better one. But that would miss my overarching point: the entire exercise is misguided. Scholarly impact—a judgment that depends upon various concrete and inchoate factors—cannot be measured in terms refined enough to produce a valid ordinal ranking. We can make broad brush assessments (supported by relevant data) about the superior productivity of certain law faculties in comparison to others, but little more can be said.

Legal academics excoriate US News for its ranking. The right course of action, it seems to me, is not to construct an alternative ranking of law faculties—particularly when every such ranking will suffer from significant distortions—but instead to eschew the ranking enterprise altogether. An ordinal ranking of law faculties is particularly inapt (US News ranks law schools, not law faculties as such).

Alas, the new ranking will have predictable consequences, some of them unseemly or downright pernicious. Soon, a number of law school websites and promotional mailings will proudly tout their impressive “Scholarly Impact” ranking. Deans will brainstorm ways to increase citation counts on their faculty (easy tips: hire more constitutional law scholars; encourage your faculty to liberally cite colleagues or thank them by name). The citation counts of potential hires will be considered in lateral hiring decisions. Law schools will be forced to pay a premium to keep high citation professors on their roster for the purposes of the count (even if in name and paycheck only); more of these folks will get simultaneous appointments on two faculties (both schools are credited with the full amount of citations). Professors will point to their high citation count as grounds for a bigger raise (Deans will use low citation counts to justify a lower raise). And so on.

The authors might protest that they cannot be blamed for misuses of their ranking system, but that hardly matters in the intensely competitive environment of legal academia, which puts pressure on law schools to maximize every measure that factors into anything that bears the dreaded title “Ranking.”

As the collective over-emphasis on citations gathers momentum, it will be too late to remind ourselves that the “scholarly impact” ranking has no more merit than a superficial beauty contest that disproportionally isolates on a couple of flashy features. The ranking will have a negative impact on legal academia simply because it exists and cannot be ignored.

The authors are not doing something better than US News—they are repeating its error of producing a ranking of dubious validity without heed to its negative consequences. The harm inflicted on law schools by US News will be compounded by an additional set of harms that follow from this new “scholarly impact” ranking. This time, however, the wounds will be self-inflicted because law professors are constructing and promoting the new ranking (lending it a patina of credibility).
Posted 11:00 AM by Brian Tamanaha [link]