Leiter Reports: A Philosophy Blog

News and views about philosophy, the academic profession, academic freedom, intellectual culture, and other topics. The world’s most popular philosophy blog, since 2003.

  1. Fool's avatar
  2. Santa Monica's avatar
  3. Charles Bakker's avatar
  4. Matty Silverstein's avatar
  5. Jason's avatar
  6. Nathan Meyvis's avatar
  7. Stefan Sciaraffa's avatar

    The McMaster Department of Philosophy has now put together the following notice commemorating Barry: Barry Allen: A Philosophical Life Barry…

How do journal editors pick referees?

Brian Leiter

A philosopher writes:

I'm a young philosopher just starting to publish and referee papers. 

It's becoming clear that different journals select referees in very different ways. Some seem to go out of their way to select referees who are knowledgeable about the specific topic of the paper. Others seem to pick referees from among the authors cited in the paper. 

Others seem to simply pick referees who work in the same area of philosophy as the one to which the paper belongs.

It'd be nice to know how common each of these procedures is, and perhaps which journals employ which procedures. The way I'd frame a paper for submission would really depend on whether I thought it'd be read by others working on the topic, or merely others working in the same general area of philosophy. Anyway, if any editors, referees, or paper-submitters out there would be willing to weigh in on this, I'd find that very helpful. Thanks!

Any comments from current or former journal editors on this subject?

Leave a Reply to Jamie Dreier Cancel reply

Your email address will not be published. Required fields are marked *

16 responses to “How do journal editors pick referees?”

  1. Jamie Dreier

    At ETHICS, the editor assigns each paper to an associate editor, and the associate editors assign referees, so the criteria for selecting referees won't be uniform.

    I try to pick referees who are particularly knowledgeable about the topic, but it’s not always possible. There are certain topics that have attracted a *lot* of submissions recently, and at ETHICS we don’t go back to a referee that we’ve recently used, so I run out of ‘super-specialists’. Then I’ll just get as close as I can; I can always get closer than just Metaethics, but maybe not as close as The Superdupervenience of the Virtues on Intrinsic Natural Properties. (I don’t particularly try to pick a cited author, but I would not rule out a referee just because she was cited.)

    Reply
  2. Rebecca Kukla

    I'm the co-editor of a (respectable but not top-tier) journal. I suspect that what I am about to say generalizes, but I have no evidence that it does: Here's how I pick referees – I desperately try to talk people who are remotely qualified to assess a paper and have no blatant conflicts of interest in doing so. Finding referees is a difficult and frustrating process. Most people say 'no' and many people do such an irresponsible job (late reviews, nonexistent reviews, irresponsibly brief or emotional reviews, etc.) that I can't use them again. I am not normally in a position to do anything remotely like impose a detailed set of criteria – I'm just happy when someone competent agrees to help out. Overall I see it as a minus, though not a deal-breaker, if the paper being reviewed cites the referee more than passingly, as the referee is unlikely to be impartial when it comes to either the prospect of seeing a discussion of her work in print, or any criticisms leveled against it.

    Reply
  3. Larry Laudan

    As a former editor who has chatted through this issue with numerous editors over the years, I'd say that you are looking for clear rules in a largely anarchic world. Practices vary between journals. Even for a given journal, refereeing conventions will depend on such factors as whether the editor's rolodex of reliable referees already includes specialists in the area of the paper, whether the paper is very technical or generally accessible and myriad other factors.

    More importantly, I'd urge the author NOT to do what he proposes doing, viz., "fram[ing] a paper for submission" to a specific journal. One is, after all, chiefly writing the paper for oneself. One does not say "I want to publish a paper in journal X" and then try to produce one tailor-made for what one (probably mistakenly) supposes to be their refereeing specifications. The order of events should be the other way round. Write a paper so that it does justice to your sense of the problem and its solution. Then, and only then, do you ask yourself the practical question: "Where should I submit it?" (Obviously, one must follow the style rules for a given journal but I would advise against allowing one's half-baked hunches about a journal's refereeing practices to play any role in 'framing' the paper.)

    Reply
  4. Kai von Fintel

    As editor of Semantics & Pragmatics (http://semprag.org), I would like to echo what Larry Lauden said: it is a truly bad idea to write your paper with reviewers in mind as the prospective audience. If you're not planning to submit to a truly niche journal (The Journal of Yet Another Counterexample to Modus Ponens), you are writing for a more or less generalist readership. The reviewers and editors are gatekeepers but what they're looking for is good work that is appropriate for whoever reads the journal.

    As for how we choose reviewers: we try to find people who are competent in the area(s) of the paper in question; we have a database of reviewers with research interest that we can search; we look at the intellectual environment of the paper (citations do lend help with that); we use our own expertise and knowledge of the field. We always try to assemble a set of 2-3 reviewers who taken together can give us a balanced assessment. There may be one reviewer with high tech competence, one with deep empirical knowledge, one with appreciation for foundational connections.

    Reply
  5. Marc Bobro

    Rebecca Kukla says that finding referees is a "difficult and frustrating process." I wonder if editors aren't making their task more difficult and frustrating than it needs to be. Here's what I mean. Are professors at "lesser" colleges being ignored? I've published in a number of good to excellent journals and have *never* been asked to referee for a journal. (Oddly, I have been asked to referee foreign dissertations.) And I would love to referee. Is the problem that I teach at a community college?

    Reply
  6. Lee Braver

    I want to chime on Marc's point. The few times I have been asked to referee, I turned in a detailed, fair (to my mind, at least) report within days. Yet I've not been asked again. It's always possible that my reports were bad in some way that I couldn't detect, of course, but I can't help but find that unlikely. Surely, they didn't dip below competent.

    Reply
  7. Eric

    I'm not an editor or a referee, but I can relay an experience of mine that might offer some explanation of one version of the process. I submitted a paper to a mid-tier journal A on a somewhat new topic with a fairly small set of existing papers. My paper was assigned to a referee who I suspect was one of the authors whose work I was commenting on in my paper. My paper was a revise-and-resubmit. A few months later, I was contacted out of the blue by a different mid-tier journal B to referee a paper on the same topic. One of the editors of journal B was one of the authors whose work I had discussed in my paper. My assumption is that the editor of journal B was my paper's referee for journal A, and had remembered my paper even though it had not been published at the time. As a late-stage grad student, I was surprised (and somewhat flattered) that I was contacted to referee a paper at all. I had no personal relationship with the editor, and was not a regular contributor to journal B.

    Reply
  8. neil

    I edit the journal Neuroethics, which I think it is fair to say has established itself as a mid-ranking journal (we have just had our first impact factor measure: 1.364 which is very healthy). There is no single procedure I use to assign referees. Often I can think of 3 or 4 appropriate referees immediately. When I can't, or when I want a fresh perspective, I often use philpapers as a resource, searching under key words through published articles. I certainly am not biased against community college academics: to be qualified to referee, the person needs to have shown evidence of relevant expertise, and that is best demonstrated by publications. So I look for people who have published a relevant paper, preferably recently.

    Reply
  9. Rebecca Kukla

    Marc and Lee: I can promise you that prestige of school plays zero role in my mind, at least. I use who I can think of. I'm human and limited and I am certain there are tons of qualified people out there I just don't happen to have heard of, so their names won't pop into my head.

    Another thing I do, when I can't really think of good reviewers, is pop a few keywords into google scholar and look for articles on similar topics that have been cited a lot of times, so if you've been cited a lot, you're more likely to be contacted by me, regardless of where you happen to teach.

    I often don't even know where people work until I google them in order to find their contact information, and this is true whether I just think of them myself or find them on google scholar.

    Reply
  10. David Beard

    There's some commonplace about being able to always get two of three: cheap, fast, and good. Cheap doesn't apply because we still give our labor away, so let me suggest "expertise, speed, and quality" as the hallmarks, and that as an editor of a midrange communication journal, my strategies were usually:

    1. Expertise: one person centrally cited in the paper, thought not if the work was a refutation of the cited person. If you build on someone else's work, you might get read by that person; if you refuted them, no way.
    2. Quality: Some reviewers just do a better job reviewing. The Expert might send a "yea/nay/maybe" and a paragraph. The Quality reviewer would comb through the paper closely. Quality will give comments not just on claims but on style and argument.
    3. Speed: When #1 or #2 drops the ball, I turn to reliable reviewer #3, who may not have expertise except broadly in the area, but who will turn it around in a weekend when #1 or #2 took two months not to do it.

    Reply
  11. Kevin

    I teach at a university that, from the general perspective of the guild, belongs to a very low ier. I have a 4-4 teaching load and the university has a religious orientation that many readers of this blog would find objectionable. But knave nonetheless been able to review, on average, at least 5 journal articles and 1 book manuscript per year over the past 5 or so years. I am quite certain that my first few requests to referee had, as their proximate cause, my mentor's turning down the invitation and suggesting me in her place. And most times when a potential referee turns down an invitation, he/she is asked for suggestions. So networking continues to be a way for getting requests. But I also probably goes without saying that publishing on a topic is perhaps the best way of being asked to referee an article on that topic or a closely related one.

    Reply
  12. Thom Brooks

    I am editor and founder of the Journal of Moral Philosophy. Selecting appropriate referees can be a major challenge. I will often research possible referees by using databases, such as The Philosopher's Index and Thomson Reuters, and often read the work of referees (if I haven't already) before inviting them to referee for the JMP. This process takes time, but I think it's important to get it right. On the whole, I think it has worked well and the JMP has benefitted as a result. But it does take time (a point worth repeating).

    Journal editing is a labour of love. Or so I try to remind myself when searching for referees or reading reviews at my desk on a sunny day…

    Reply
  13. a

    From the comments above there seems to be a classic market mismatch. Editors need reviewers; some people who would like to review are not even being consider. Perhaps an internet database could be constructed of people who would like and be willing to review, along with their specialties and papers they have written, which editors could turn to first if they don't have anyone who leaps into mind? I'd suggest PhilPapers as a logical place for this database, but it could be anywhere…

    Reply
  14. Lee Braver

    Rebecca,

    I should have been more specific. The point I wanted to chime with was my willingness to perform a duty often described by those doing it as onerous, and by those needing it as hard to find, while my door goes unknocked. Even when I have offered, or have reviewed and expressed my willingness to do so again, I've rarely been asked.

    Again, the possibility that I did a poor job cannot be ruled out–esp. for the couple of journals I reviewed for–but I certainly returned detailed reports quickly, which I tried to fill with constructive suggestions rather than just criticism after criticism when I found the article in question wanting. In one case, when I positively reviewed a ms for a well-known scholar, he was extremely grateful for the suggestions I made. (Since it was positive, I gave permission to disclose my identity to him).

    I think that Kevin got it right when he pointed to networking. I don't travel in the circles that would introduce me to editors, which would lead to being on more lists of potential referees. This is not identical with where one teaches, of course, but it does overlap with it, since the better schools bring in big name presenters, visiting scholars, etc. I also have focused more on books than articles, which may have given my work a lower profile.

    In any case, the database idea sounds great. This might be especially helpful for recently graduated Ph.D.'s looking to learn more about the publishing process but who have not yet established a record of scholarship.

    Reply
  15. Rebecca Kukla

    Lee and 'a': I agree. I try to pass on referee requests I am too busy to do to my good grad students and to people I know and respect who are quite junior, but I only know the people I know, as it were. I wouldn't use someone off a database unless I could at least read one or two papers by them, but some sort of networking tool would be to everyone's benefit.

    Reply
  16. Henry S. Richardson

    More and more journals now–including Ethics, which I edit–are using web-based manuscript management systems, and various of these have a feature that allows editors to "grade" reviewers or otherwise record remarks about the quality of reviews. Of course, this would be extra work for the editors, and I've had little success persuading our editors to use it, as we're rather overwhelmed with the flow of submissions; but if journals would use such a system, then it would at least help prevent editors from losing track of good reviewers like Lee. And, being a database built up by the editors, it would naturally be more trusted by the editors than would a more general database, thus addressing Rebecca's worry about the latter. I have a sense, though, that these utilities for searching for potential reviewers are rather cumbersome. I wonder: has any philosophy journal implemented such a system?

    Reply

Designed with WordPress