Someone should really do this again, since U.S. News reputational surveys are done so poorly.
The Survey and the Method
Between March 3 and March 21, 2003, more than 150 leading legal scholars from around the nation completed the most thorough evaluation of American law faculty quality ever undertaken. Scholars were invited to participate in the EQR survey based on the following criteria:
- Only active and distinguished scholars were invited. These are the scholars most likely to have informed opinions about faculty quality.
- Multiple faculty from every school evaluated were invited
- Diversity in terms of seniority was sought in the evaluator pool.
- Diversity in terms of fields and approaches was sought in the evaluator pool.
Evaluators were not permitted to evaluate either their own institution or the institution from which they had received the highest law degree.
The quality of evaluators in this survey is unparalleled: it includes the President and President-elect of the Association of American Law Schools; a dozen members of the American Academy of Arts & Sciences, the nation’s most prestigious learned society; dozens of the most frequently cited legal scholars in numerous fields; and leading figures, junior and senior, in corporate law, criminal law, health law, constitutional law, jurisprudence, international law, comparative law, legal history, feminist legal theory, and many other fields. The complete list of evaluators appears in Appendix A.
Evaluators were presented with faculty lists for 2003-04, based on the best present knowledge about who is teaching where next academic year. Many important faculty moves for 2003-04 were accounted for in the faculty lists evaluators received, including, for example:
Stuart Benjamin’s move from Texas to Duke
Richard Delgado’s move from Colorado to Pittsburgh
John Duffy’s move from William & Mary to George Washington
Merritt Fox’s move from Michigan to Columbia
Elizabeth Garrett’s move from Chicago to Southern California
Deborah Malamud’s move from Michigan to NYU
Ronald Mann’s move from Michigan to Texas
Andrei Marmor’s (part-time) move from Israel to Southern California.
Hiroshi Motomura’s move from Colorado to North Carolina
Robert Peroni’s move from George Washington to Texas
Michael Perry’s move from Wake Forest to Emory
Stephen Perry’s move from Penn to NYU
Arti Rai’s move from Penn to Duke
Jim Rossi’s move from North Carolina to Florida State
Chris Sanchirico’s move from Virginia to Penn
But some faculty moves transpired subsequent to the survey. For example:
Ellen Deason is moving from Illinois to Ohio State.
Martha Fineman is moving from Cornell to Emory.
Thomas Gallanis is moving from Ohio State to Washington & Lee
Larry Garvin is moving from Florida State to Ohio State.
Jack Goldsmith is moving from Chicago to Virginia.
Michael Heise is moving from Case Western to Cornell.
Thomas Merrill is moving from Northwestern to Columbia.
Dale Oesterle is moving from Colorado to Ohio State.
Robert Post is moving from Berkeley to Yale
Peter Shane is moving from Carnegie-Mellon to Ohio State.
William Simon is moving from Stanford to Columbia
Aviam Soifer is moving from Boston College to University of Hawaii (to become Dean)
Where subsequent moves were likely to have affected the evaluations in specialized ares of legal scholarship, they are noted below. Ordinarily, the addition or loss of one faculty member is not likely to have a significant effect on a school’s overall score, but there can be exceptions, especially for prominent senior figures moving to or from smaller faculties (e.g., Fineman). Ohio State, which had multiple hires, likely would have picked up .1 or .2 in its overall score.
69 schools that had some plausible claim to being in “the top 40” in terms of faculty quality were included in the survey. The complete instructions to evaluators appears in Appendix B and the full list of schools evaluated appears in Appendix C. Note that this is not supposed to be a list of the “top 69” schools; plainly other schools could have been included in the mix, but as it turns out the borderline contenders for inclusion did not come close to the top 40 in the survey.
The EQR survey is confined to “the top 40” faculties overall precisely because there is reason to think that active legal scholars, who are engaged in reading and producing scholarship, are in a good position to give a meaningful evaluation of at least the top 40 law faculties. In addition, the “top 20” in eleven different specialty areas are also included. There is reason to think that the 69 schools surveyed would dominate the “top 20” in the specialty areas. (Note: Deans may request complete results for their school, where those results are not available here; e-mail bleiter@mail.law.utexas.edu.)
Evaluators were presented with faculty lists identified only by number; no school names were provided. This was an attempt to avoid the “halo” effect. Of course, most evaluators could identify the schools, but it clearly makes a significant difference to begin an evaluation by reading a list of faculty names and then saying, “ah-ha, this is Harvard,” as opposed to starting an evaluation by seeing “Harvard.” As one evaluator remarked:
“Giving the schools numbers instead of names is useful; it reinforces a strong message [even if] it does not cloak the identity of the schools.” And as another wrote: “There’s a surprising difference between my perception of faculty quality [based on school name] and how I rank them when I am faced with a list of faculty members.”
The Rankings
Evaluators were asked to evaluate schools on a scale of 1 (weak) to 5 (excellent). Evaluators plainly had different centers of gravity in their rankings, with some giving few 5s, while others gave hardly any 1s. In order to guard against strategic and utterly idioscynratic voting, “outlier” scores were dropped. Outlier scores were defined as follows: the distribution of 5s, 4s, 3s, 2s, and 1s for each category was charted; where there were not more than two 5s or two 1s in a category where the school in question had either no 4s or no 2s, the scores were deemed to be “outliers” and dropped. (So, e.g., if a school in some category had received ten 5s, seventy 4s, and sixty 3s, no 2s, but then one or two 1s, the 1s would be dropped.) This resulted in approximately a dozen scores of 1 being dropped out of several thousand scores entered. This indicates, of course, that evaluators performed their tasks honorably. The mean and median scores printed below are after the outlier scores were dropped.
The size of each faculty is also listed in parentheses following the school name. There is some controversy over the impact of school size in a survey like this. Some argue–plausibly in my view–that larger faculties have an advantage, because it is more likely that an evaluator will know the work of someone on the faculty. Others contend that large schools deserve this advantage: if a school has more faculty doing work known to more scholars, doesn’t that mean it’s a stronger faculty? Still others deny that large schools get any advantage, and, indeed, that they are at a disadvantage. As one evaluator wrote: “If I had only ever heard of one person on a faculty, one interpretation is that that was a weak faculty.” Thus, a large faculty with only one or two names known to an evaluator may be at risk for faring worse. In any case, students can see for themselves the differences in faculty size, below.
Other biases work their way into surveys of opinion about faculty quality. Schools on the two coasts are usually at an advantage in opinion surveys, because schools are more tightly clustered, geographically, and faculty tend to know each other better, both professionally and socially. Unsurprisingly, evaluators score faculties strong in their areas of expertise more highly; although a mix of evaluators (in terms of expertise) participated, it is clear that a slightly different mix would have some impact on the results. Since it is reasonable to suppose that changes in the mix of evaluators (even equally qualified evaluators) could lead to changes of .1 in either direction, I list runners-up both for the overall rankings and in the specialty rankings.
Runners-Up for the Top 40:
| School (number of faculty) | Mean | Median |
| American University (51) | 2.4 | 2.5 |
| Brooklyn Law School (43) | 2.4 | 2.5 |
| Rutgers University, Newark (39) | 2.4 | 2.5 |
| University of Florida, Gainesville (59) | 2.4 | 2.5 |
| University of Notre Dame (37) | 2.4 | 2.5 |
(Note: in 2003-04, the UT Austin faculty included as full-time faculty Mitchell Berman, Philip Bobbitt, Douglas Laycock, Sanford Levinson, Lawrence Sager, and Ernest Young, among other luminaries. (Only Levinson and Sager remain, alas.)
List of evaluators
292 evaluators from around the nation received invitations to participate in the survey, and 51% accepted the invitation and completed the surveys. Another 2% tried to complete the surveys, but encountered technical problems with the on-line survey that we were unable to resolve in time. Roughly another 11% expressed an interest in participating, but an inability to do so during the indicated time period. (It is fair to say that certain rather more weighty matters going on in the world during the same time period depressed the response rate somewhat.) Not quite 2% expressed an unwillingness to participate, and the remaining third were not heard from.
| Evaluator | School |
| Kenneth Abraham | University of Virginia |
| Matthew Adler | University of Pennsylvania |
| Peter Alces | College of William & Mary |
| Gregory Alexander | Cornell University |
| Larry Alexander | University of San Diego |
| Ronald J. Allen | Northwestern University |
| Vikram Amar | University of California, Hastings |
| Keith Aoki | University of Oregon |
| Stephen Bainbridge | University of California, Los Angeles |
| Lynn Baker | University of Texas, Austin |
| Milner Ball | University of Georgia |
| Randy Barnett | Boston University |
| David Bederman | Emory University |
| Margaret Berger | Brooklyn Law School |
| David Bernstein | George Mason University |
| Lisa Bernstein | University of Chicago |
| Guyora Binder | State University of New York, Buffalo |
| Brian Bix | University of Minnesota, Twin Cities |
| Alfred Brophy | University of Alabama |
| James Brudney | Ohio State University |
| Harold Bruff | University of Colorado, Boulder (also Dean) |
| Ronald Cass | Boston University (also Dean) |
| Robert Chang | Loyola Law School, Los Angeles |
| R. Alta Charo | University of Wisconsin, Madison |
| Jim Chen | University of Minnesota, Twin Cities |
| Gabriel Chin | University of Arizona |
| John C. Coffee, Jr. | Columbia University |
| Jules Coleman | Yale University |
| Hanoch Dagan | University of Michigan, Ann Arbor |
| Anne Dailey | University of Connecticut |
| Kenneth Dau-Schmidt | Indiana University, Bloomington |
| Adrienne Davis | University of North Carolina, Chapel Hill |
| Richard Delgado | University of Pittsburgh |
| Neal Devins | College of William & Mary |
| Michael Dorf | Columbia University |
| Joshua Dressler | Ohio State University |
| Donald Dripps | University of Minnesota, Twin Cities |
| Markus Dubber | State University of New York, Buffalo |
| Ira Ellman | Arizona State University |
| Daniel Farber | University of California, Berkeley |
| Noah Feldman | New York University |
| Lee Anne Fennell | University of Texas, Austin |
| David Fischer | University of Missouri, Columbia |
| Richard Michael Fischl | University of Miami |
| James Fleming | Fordham University |
| Eric Freedman | Hofstra University |
| Richard Freer | Emory University |
| Barry Friedman | New York University |
| Philip Frickey | University of California, Berkeley |
| Richard Garnett | University of Notre Dame |
| Mark Gergen | University of Texas, Austin |
| Clayton Gillette | New York University |
| John Goldberg | Vanderbilt University |
| Risa Goluboff | University of Virginia |
| Jeffrey Gordon | Columbia University |
| Robert Gordon | Yale University |
| Mark Grady | George Mason University (also Dean) |
| Abner Greene | Fordham University |
| Stephen Griffin | Tulane University |
| Ian Haney Lopez | University of California, Berkeley |
| David Herring | University of Pittsburgh (also Dean) |
| Michael Herz | Cardozo Law School/Yeshiva University |
| Herbert Hovenkamp | University of Iowa |
| Heidi Hurd | University of Illinois, Urbana-Champaign (also Dean) |
| Alan Hyde | Rutgers University, Newark |
| Nancy King | Vanderbilt University |
| Thomas Kohler | Boston College |
| Donald Kommers | University of Notre Dame |
| Russell Korobkin | University of California, Los Angeles |
| Jody S. Kraus | University of Virginia |
| Harold Krent | Chicago-Kent College of Law |
| Andrew Kull | Boston University |
| Christopher L. Kutz | University of California, Berkeley |
| Donald Langevoort | Georgetown University |
| Lewis LaRue | Washington & Lee University |
| Gary Lawson | Boston University |
| Douglas Laycock | University of Texas, Austin |
| Leandra Lederman | George Mason University |
| Brian Leiter | University of Texas, Austin |
| Mark A. Lemley | University of California, Berkeley |
| Sanford Levinson | University of Texas, Austin |
| Saul Levmore | University of Chicago (also Dean) |
| Michael Livingston | Rutgers University, Camden |
| Kyle Logue | University of Michigan, Ann Arbor |
| Ira Lupu | George Washington University |
| Jonathan Macey | Cornell University |
| William P. Marshall | University of North Carolina, Chapel Hill |
| Toni Massaro | University of Arizona (also Dean) |
| Calvin Massey | University of California, Hastings |
| Richard McAdams | University of Illinois, Urbana-Champaign |
| Geoffrey Miller | New York University |
| David Millon | Washington & Lee University |
| Robert Mnookin | Harvard University |
| Michael S. Moore | University of Illinois, Urbana-Champaign |
| Stephen Morse | University of Pennsylvania |
| Andrew Morriss | Case Western Reserve University |
| Christopher Mueller | University of Colorado, Boulder |
| Stephen R. Munzer | University of California, Los Angeles |
| Nell Newton | University of Connecticut (also Dean) |
| Gene Nicol, Jr. | University of North Carolina, Chapel Hill (also Dean) |
| Mark Osiel | University of Iowa |
| Dennis Patterson | Rutgers University, Camden |
| Jordan Paust | University of Houston |
| Stephen Perry | New York University |
| Christopher J. Peters | Wayne State University |
| Richard H. Pildes | New York University |
| Daniel Polsby | George Mason University |
| Richard A. Posner | University of Chicago and U.S. Court of Appeals for the Seventh Circuit |
| Peter Quint | University of Maryland |
| Nancy Rapoport | University of Houston (also Dean) |
| Steven Ratner | University of Texas, Austin |
| Jerome H. Reichman | Duke University |
| William Reynolds | University of Maryland |
| Annelise Riles | Cornell University |
| Daniel Rodriguez | University of San Diego (also Dean) |
| Roberta Romano | Yale University |
| Jim Rossi | Florida State University |
| J.B. Ruhl | Florida State University |
| Teemu Ruskola | American University |
| Brett Scharffs | Brigham Young University |
| Anthony Sebok | Brooklyn Law School |
| Mark Seidenfeld | Florida State University |
| Michael Selmi | George Washington University |
| George Shepard | Emory University |
| Emily Sherwin | Cornell University |
| Jonathan Simon | University of Miami |
| Christopher Slobogin | University of Florida, Gainesville |
| Michael Solimine | University of Cincinnati |
| Larry Solum | Loyola Law School, Los Angeles |
| Peter Spiro | Hofstra University |
| Matthew Spitzer | University of Southern California (also Dean) |
| Kate Stith | Yale University |
| Lynn Stout | University of California, Los Angeles |
| Kent Syverud | Vanderbilt University (also Dean) |
| Eric Talley | University of Southern California |
| A. Dan Tarlock | Chicago-Kent College of Law |
| George Thomas III | Rutgers University, Newark |
| Joseph Tomain | University of Cincinnati (also Dean) |
| Gerald Torres | University of Texas, Austin |
| William Treanor | Fordham University (also Dean) |
| Mark Tushnet | Georgetown University |
| Eugene Volokh | University of California, Los Angeles |
| Elizabeth Warren | Harvard University |
| Donald Weidner | Florida State University (also Dean) |
| Mark Weiner | Rutgers University, Newark |
| Michael Wells | University of Georgia |
| Robin West | Georgetown University |
| Tobias Wolff | University of California, Davis |
| Barbara Woodhouse | University of Florida, Gainesville |
| Fred Zacharias | University of San Diego |
| Benjamin Zipursky | Fordham University |
I am shocked and so deeply sorry to hear about this. My thoughts are with Zafra and his family. I…