Leiter Reports: A Philosophy Blog

News and views about philosophy, the academic profession, academic freedom, intellectual culture, and other topics. The world’s most popular philosophy blog, since 2003.

  1. Jamie Kelly's avatar
  2. Daniel Fidel Ferrer's avatar
  3. Ryan Mitchell Wittingslow's avatar
  4. Panu Raatikainen's avatar
  5. Jason Stanley's avatar
  6. Daniel Greco's avatar
  7. Nobody's avatar

How much effort/time should a journal referee expend on a paper?

Brian Leiter

A senior philosopher elsewhere writes:

Like many readers of this blog, I am frequently asked to referee journal submissions.  Though I recommend acceptance only very rarely, I feel duty-bound to work as hard on something I agree to referee as I would on something I read for my own scholarship, trying to puzzle through the details of the central arguments and to figure out exactly where those arguments go wrong (when they do).  As a result, I would guess that I usually spend 3 hours on a paper I'm refereeing, though often enough it is more than that, and I generally find myself with about two single-spaced pages worth of comments.  I have never served on an editorial board, and so I have never been in a position to see a large number of referee reports.  Reports that I've received on my own submissions suggest a wide variation in the time and effort other people put into refereeing.  I'd be interested to hear from journal editors and other readers about what they think constitutes a conscientious effort.

My sense is this sounds like a responsible refereeing effort.  But it would be useful to hear what others think.

 

Leave a Reply

Your email address will not be published. Required fields are marked *

17 responses to “How much effort/time should a journal referee expend on a paper?”

  1. Sara L. Uckelman

    As a member of two editorial boards, I've seen the range of reports from very short to very long, and there isn't necessarily a correlation between usefulness and length (though rarely has a long one not been useful).

    For my own part, I read the paper over once, writing lots of comments as I do so, and then I read it over again as I type the comments up into my report. This generally takes about an afternoon, and my reports are 2-6 pages long (depending on how many typos/grammar errors/etc. there are!).

    Reply
  2. Steven Hales

    I've never been a journal editor, but I'm regularly asked to referee. I'd say the effort Senior Philosopher Elsewhere puts in is commensurate with what I do. I probably write an average 1000 words in a report, although this does vary. I have two articles on my desk right now to referee: one is 50 pages and the other is 17. They are obviously going to require different amounts of effort.

    Reply
  3. Out to pasture

    Here's a brusque (but I hope not brutal) headline thought. Referee's reports for journals are written for editors. They should offer a judgement about whether the paper is worth publishing. They are not, primarily, tutorial reports for authors. If the judgement is negative, then it is a work of supererogation to spell out the reason in any detail: a couple of lines for the editor's eye is enough. The editor should trust your judgement if s/he has chosen you to report. It is nice if you have the time to say more, but there is no need. Indeed you can perfectly properly stop reading as soon as it becomes clear enough that the paper isn't up to the required standards, for whatever reason. Spend the time instead on other reports, making your positive ones as useful as possible for the journal by offering suggestions for improvement, if you have them.

    I wouldn't always have put things so brusquely. Once upon a long time ago, philosophers could be working in considerable isolation from each other — perhaps being the only metaphysician, philosopher of science, or whatever in their department, very many miles from a colleague with similar interests (and conferences where they might get to chat over ideas were a very occasional treat). Back in the day, extended comments on their efforts from journal referees might be some of the only feedback they got. The quasi-tutorial report, especially for those starting out, had an important role. But that world has gone. We have all kinds of ways of networking and getting preliminary feedback on our ideas from circulated drafts to our contacts. And therefore — or so it seems to me — the role of journal referees has become more sharply defined. It is to give advice for editors, not authors. Anything else is optional.

    That's the line I took a while back, when an editor. It at least helped promote fast reports.

    Reply
  4. Brian Leiter

    I'll just say that I disagree with #3, but I wonder what others think about this position?

    Reply
  5. Christopher Morris

    I now turn down a lot of requests to review articles for a very simple reason: it takes me too long to write comments for authors. I normally can read a paper in a couple of hours, more for something approaching 10,000 words or more. But writing comments takes me hours, sometimes most of a day (I'm slow). Given other demands on my time, I rarely finish the job in a timely fashion.

    So I have to say I'm much more likely to accept an invitation to review a submission if the journal requests only a note to the editors, as per Out to Pasture's suggestion. I should imagine that this suggestion would also considerably shorten the time submitters now need to wait to learn the fate of their essays.

    Reply
  6. Friend of OTP

    Like Out to pasture, I believe that my referee reports are written for the editor, first and foremost. They are to aid her in making a decision about the manuscript under consideration. But I also believe that the referee must justify her assessment of the manuscript. Consequently, a well written referee's report will provide valuable insight for the author, if they seek to improve the paper, even if it is rejected and will need to be submitted to another journal. But I do not see my job as one of correcting every grammatical error, or catching every spelling mistake. I am responsible for evaluating the argument, and the way in which the author presents the views of others.
    I serve on an editorial board for a science journal. I make the final calls on manuscripts that I handle. I count on the referees' reports to give me insight into the quality of the paper under review. Useful reports tell me what the referee thinks about the paper, but also why it should be accepted or rejected.
    I am alarmed at how many very weak papers are submitted to philosophy journals. The arguments are quite thin. There is little knowledge of the existing literature on the topic. They misrepresent the view of others, and thus often attack a straw-philosopher. People need to realize that they should only submit a paper when they have developed the argument, and prepared the manuscript in a professional manner. That is why conference presentations are so important. Conferences provide useful venues for improving a paper, getting them ready for publication in a journal.
    One does not see that type of thing in the science journal I work on. There is a minimal level of competence that a paper must exhibit; and there is a general structure to science papers that ensures that papers submitted generally meet that minimal standard. I still reject numerous papers for this science journal. But the grounds for rejection are almost always strictly due to the unsupported inferences drawn from the evidence.

    Reply
  7. Jerry Green

    One more vote on the OP side: I typically spend 3-4 hours on a report (maybe longer if you count the initial read-through to see if I feel competent to ref in the first place). Mine tend to come in at 1500-2000 words, but I use a template where I say something about multiple issues (scholarship, contribution, etc), so that tends to add to the word count a bit.

    I'll also add my disagreement with Out to pasture: I think that (i) the author is an important part of your audience, and (ii) even if you only write for the editor, they should be able to see the reasoning behind your judgment. If there are multiple referees, for instance, it will matter how their interpretations correspond. And given how many negligent referee reports I've seen/heard about, I don't think it would be responsible of an editor to trust that a review of only a few sentences reflects a conscientious and fair effort (yes, I know lots of referees work hard, but its the bad apples that ruin it for the rest of us).

    But Out to pasture is at least right about fast reports. I guess there's a compromise that 'if the report is short, it had better be real fast too'.

    Reply
  8. Russell Blackford

    When I'm wearing my journal editor hat, I don't usually want to digest pages of analysis but I do want to know the case for accepting or rejecting, or for asking for specified revisions. A concisely written page is often enough, and it can be even shorter if the reviewer thinks it's a clear-cut decision. When I'm asked to review a paper I generally end up writing about a page, but again sometimes much less if I think the case for acceptance or rejection is clear. The more difficult ones to write are recommendations to revise and resubmit – those sometimes need to be quite long. Because I often find myself providing a report via an online form that keeps a record of the process in the reviewer area of my account, I can often, later on, check out other people's (anonymous) reviews and the editorial decision. Most reviews that I see by others are very brief – only a paragraph or so. Reviews that I receive as an editor vary greatly in length from a couple of sentences to several pages.

    I find that it usually takes me an afternoon or an evening to absorb a paper's argument and write a report.

    Reply
  9. David Duffy

    In many fields (consider mathematics), most papers are accepted after a revision addressing the reviewer commments, and the contributions of these to improving the manuscript are often acknowledged formally in the final publication. The open publication approach showing all the editor-reviwer-author correspondence is becoming more common – I think it's on balance a good thing. I too write ~1000 words (scientific reviews). Short amusing putdowns are entertaining, but not very useful.

    Reply
  10. Daniel Kaufman

    I was a Managing Editor for years, and I completely disagree with (3). For one thing, there is the assumption that everyone has access to the kind of networking described there, which is simply false. Many, many of us are working out in the sticks, in tiny departments, community colleges, departments merged with other subject areas, sometimes in quite physically isolated areas. (If you think it's as simple as just emailing Jerry Fodor or Stephen Schiffer and asking them to please give you their opinion on the paper you are working on, in Phil Mind or Phil Language, it isn't.) And for another — and much more importantly — there is the simple matter of basic decency and professionalism. Someone submits something to you and you reject it, you tell them why. Not only is this not supererogatory, anything less is just plain uncouth.

    When I referee papers, I write them in such a way so that *both* the Editor and the Author understand why I have given the verdict that I have given. Indeed, so normal and obvious does this seem to me that I found the initial question frankly a bit surprising. And don't tell me about workloads — when I was Managing Editor, there was no staff — I was it — and was Adjuncting, at the time, with a full load of classes (i.e. 4), not to mention working on my dissertation. People in other professions can only dream of the amount of time we have off, not to mention the flexibility of that time, and to suggest that we can't give a person a decent explanation of why their paper was rejected, because we're too busy reflects a real lack of consciousness about just how much people in other professions work.

    Reply
  11. Marcus Arvan

    If my first, second, and third-hand experience are any indication, a decent proportion of philosophy journal referees appear to subscribe to Out to pasture's general line of thought. I have not only had papers rejected with one or two lines to the effect of, "The arguments are no good", without any substantiation of the reviewer's judgment. I have also had numerous collegagues in the discipline–including some people in my social media feeds recently–recount similar experiences: as having to wait for anywhere from 3 months to over a year to get a rejection consisting of a couple sentences that amount to little more than, "Trust me, this paper is not publishable–though I cannot be bothered to give you any explanation why." Interestingly, although I do not have much experience in other fields, I do have some first and second-hand experience with one other field: psychology. I have submitted papers to psychology journals myself, and my spouse is a researcher in Industrial-Organizational Psychology. And, although my data points might not be representative, I have yet to come across a single instance of this kind of reviewer behavior there. Even the bad papers I submitted to psychology journals were met with clear explanations of what was wrong–in some cases, explanations by a reviewer, in other cases a brief explanation by the editors. And every referree report my spouse has shown me with her papers is similar: there is always a real explanation–a justification–of the referree's recommendation.

    Of course, these differences between fields might be justified. Maybe philosophers have special reasons–special qualifications or abilities–that should exempt them from having to justify their editorial recommendations. Or maybe there are more general reasons to think referees shouldn't have to give detailed recommendations. This seems to be Out to pasture's thought: that in soliciting a referee's recommendation to begin with (especially, let's say, if the person has a good philosophical reportation), philosophy editors have grounds to trust the judgment of that referee. Yet this seems to me to fly in the face of everything we know, empirically–and we know a lot–about the pernicious effects of tacit biases. People tend to think that they are "objective, impartial" judges of things–and it seems, in philosophy, that we judge "the arguments on their merits." But there are so many empirical reasons to doubt this. People tend to think that they do not judge people differentially on the basis of race, gender, social class, etc.–yet, time and again, empirical research shows they do. Similarly, although a given referee might not think they judge papers in biased ways, there are all kinds of other possible ways that their judgments might be biased. For example, one recent study (http://dl.acm.org/citation.cfm?id=981001) of doubly-anonymized review showed that,

    "[A]uthors often could be identified by reviewers using a combination of the author's reference list and the referee's personal background knowledge…[identifying] authors correctly 40-45% of the time. One main motivation for double-blind review is to eliminate bias in favor of well-known authors. However, identification accuracy for authors with substantial publication history is even better (60% accuracy for the top-10% most prolific authors, 85% for authors with 100 or more prior papers)."

    Second, many papers are shared publicly–at conferences, department colloquia, etc.–in ways that can clue a reader into the author's identity. Third, there are "graduate student" ways of writing, or so I hear–features of a person's writing which can signal a person being an early-career philosopher. And so on.

    This, in brief, is why referees are obligated to give actual, detailed justifications for their recommendations, and why editors should expect them to fulfill that obligation. The point of anonymized review is (or should be) to mitigate bias, and ensure that papers are judged on their merits. Having to give an actual justification is one way of ensuring this: it is a way to (A) hold referees accountable for their recommendations, and (B) enable second and/or third parties (an editor or multiple editors) to judge whether the judger actually has a good case to make for the recommendation, or whether their recommendations are due to biases they (the referee) may or may not be aware of. I just have a hard time seeing how anything less is appropriate scholarship. Given the kinds of biases that can–and are known to–afflict human judgers, "trust me" shouldn't be considered good enough, especially when it comes with a 3-6+ month wait time.

    Reply
  12. Mohan Matthen

    Referees have a responsibility to ensure to the best of their ability that what gets published is accurate, well reasoned, and worth saying. I find it frustrating to read papers in my areas of expertise that are not all of the above, and I find it particularly frustrating that my graduate students waste a lot of time on some of these papers. Fulfilling this responsibility takes me three to four hours for a paper that is around 8,000 words long.

    As for reports, I am closer to Out to Pasture (comment 3) than to those who oppose her/him. One can certainly explain one's recommendation in a couple of paragraphs, provided that explaining is not equated with arguing the point. In any case, I don't find myself or my friends agreeing more with the more argumentative reports. On the contrary, it seems that the longer a negative comment is, the more an author will find to disagree with.

    There is a faint suggestion in one of the anti-OTP comments that one has a duty to help the authors write a good paper. (Networking is hard to find.) I would find this presumptuous: who am I to give advice to someone I don't know? In any case, I referee perhaps twenty five papers a year—though recently I have been trying to do no more than one a month—and I certainly wouldn't take on that number if I thought that this was expected of me.

    Reply
  13. Steven Hales

    By parity of reason, the longer a paper is, do you find more to disagree with? If so, then loquacious authors should hope that you are not sent their paper to referee!

    Reply
  14. Ed

    I agree with those thinking that it is not the duty of referees to help the author write a good paper. The duty is to assess it, and explain the reasons for their recommendation (rather than to justify it in full detail). In my experience this can often be done in no more than two paragraphs. For example, you can say "the paper does not address the key points made Barndom in MIE" without giving chapter and verse). I think this is what both the editor and authors need. After doing that, time permitting, I like to offer more detailed comments that may help the author, without pretending to be exhaustive. This is important, since rewriting often involves trade-offs (we all know what it feels like when reviewer 1 asks you to add something and reviewer 2 says the paper is too long).

    Some journals ask for a recommendation to the editor and encourage you to supply comments for the author, but don't insist on this. I think this makes sense. By the way, this is also how thing happen in science: reviewers will tell you if they think your experiment is bogus; it is not their duty to design better experiments for you.

    Reply
  15. Just a snarky point.

    "the paper does not address the key points made Barndom in MIE" is often roughly the level of detail, spelling and grammatical correctness, and informativeness of reports I receive. Make of this what you will.

    Reply
  16. Ed

    Touche 😉
    And yet the point stands.

    Reply
  17. Charles Pigden

    Typo or inspired invention?

    Marcus Arvan writes : In soliciting a referee's recommendation to begin with (especially, let's say, if the person has a good philosophical reportation), philosophy editors have grounds to trust the judgment of that referee.

    Is a good philosophical reportation a reputation for writing good philosophy reports?

    —–
    KEYWORDS:
    Primary Blog

    Reply

Designed with WordPress