Leiter Reports: A Philosophy Blog

News and views about philosophy, the academic profession, academic freedom, intellectual culture, and other topics. The world’s most popular philosophy blog, since 2003.

  1. LFC's avatar
  2. Rob's avatar
  3. Dan Schwarcz's avatar
  4. Milan Markovic's avatar
  5. Andrew Mamo's avatar
  6. Stefan Sciaraffa's avatar

    Please see this link for the final version of the McMaster Philosophy Department’s notice for Barry Allen. There is some…

  7. Patrick S. O'Donnell's avatar

“Reject without comment” option at journals: would philosophers want that?

Brian Leiter

A philosopher involved in journal editing writes:

I have recently run across a paper titled "Edifying Editing" [ed.-link now fixed] by a co-editor of the American Economic Review. Only some parts of the paper are relevant to philosophical journals, but those parts are interesting. Here is a modified version of an editorial policy that AER follows. I'd be interested in whether it would appeal to philosophers.

Suppose that a journal allowed authors to opt in advance for their manuscript to be reviewed on a "comments only for revision" basis. When an author opts for this treatment, the editors would instruct referees that they need not provide comments if they are recommending a flat rejection of the manuscript: comments are to be provided only by referees whose recommendation is "accept" or "revise-and-resubmit"  – that is, only if comments would be useful for ongoing consideration and/or publication of the paper. 

The purpose of selecting this option would be that negative decisions would be forthcoming more quickly, so that authors could submit their papers elsewhere. Authors would not receive comments that help them to improve their work for submission to other venues, but they would be able to move on to those venues with less delay.

Again, this treatment of a manuscript would be at the author's option. 

What say you?
Comments are open; signed comments strongly preferred.  Submit your comment only once, it may take awhile to appear.

Leave a Reply

Your email address will not be published. Required fields are marked *

42 responses to ““Reject without comment” option at journals: would philosophers want that?”

  1. Fritz Allhoff

    Great, make it so. Something just has to be done about these top journals and these review times that are spiraling out of control. Even the ones that do a good job (e.g., PPR and Nous) suffer as everyone redirects over there because these other places take forever. Maybe when stuff had to get mailed around 6+ months made sense, but it's just crazy now. There's been so much complaining about journals in the blogosphere recently; I really hope the editors are reading. (Though one premiere journal's editor recently did offe a reply that made quite little sense.)

    Reply
  2. Dennis Whitcomb

    This system might help if, in those cases where authors slate their papers for no-comments reviewing, the referees are held to shorter review times. Shorter as in like about two weeks. Referee work expands to the time you allot for it, after all.

    Reply
  3. Nikk Effingham

    Something has to be done, to be sure. I'm not sure if this is the best solution, although I suppose it can't hurt. When an article I referee is particularly dire (obviously unpublishable, terribly structured etc.) I admit the option to give no comments would be nice. But for the vast majority of papers this isn't the case – and I for one wouldn't trust myself to referee it without being able to give comments on it. After all, if I can't write down, clearly and succinctly, what about it means it should be rejected I worry that my opinion might be wrong. And, indeed in some cases, during the process of giving comments it becomes clear to me that I've misread the paper and, actually, I should revise my negative judgment.

    I'm not sure how many people feel the same, but my personal worry about ticking the 'no comments' box is that not only will I receive no comments (which is bad in and of itself) but the refereeing may not be of the best standard, and I might get rejected for utterly baseless reasons. So whilst it can't _hurt_ to include the tick box, I for one wouldn't tick it. As I guess (and I might well be wrong!) that other people will think similarly, I doubt this option will help a great deal in reducing the load. But then with the insane turn around times, and horror stories about journal submissions being commonplace, I reckon anything is worth a go.

    Reply
  4. CharlieH

    Sometimes, in working through my reasons for recommending rejection, I discover the paper's better than I thought. Also, don't authors learn more from negative feedback than from positive feedback? Maybe "Accept without comment" is a better idea, though it wouldn't cut the workload much, I suppose.

    Reply
  5. Doug Portmore

    Like Nikk, I wouldn't choose the "comments only for revision" option. First, along with Nikk, I worry that you wouldn't get the best refereeing that way. I find that, when I referee, I sometimes realize through the process of writing out the referee report that some vague objection that I initially had doesn't actually have merit on further reflection. And I don't think that referees can do their most careful thinking about submissions without writing out reports on them. Second, I worry that referees will be more inclined to reject a submission outright than to suggest revise-and-resubmit if they're on the fence between the two and realize that if they choose the latter they'll have to write out a referee report. Third, the referee reports that I've received (even in the cases of outright rejection) have been crucial to my professional development. I've learned a lot from these reports.

    Of course, from the fact that I wouldn't take the option, it doesn't follow that it would be bad to implement a policy where authors are given this option. But I have some worries about the policy itself. I worry that referees will come to resent those who don't choose the "comments only for revision" option. I worry that editors won't be getting as good advice from referees as they would otherwise get given how important writing out one's thoughts is to thinking clearly. And I worry that young philosophers who are under the pressure of a tenure clock will choose the "comments only for revision" option to the detriment of their professional development.

    I do, however, support the policy that's now in place at most journals where there is some sort of initial vetting that weeds out those papers that are clearly non-starters and don't even deserve to be passed on to referees (or, in the case of Nous, passed on to further referees).

    Reply
  6. Mohan Matthen

    I have been both an editor and a referee.

    As referee, most papers that I look at require close attention. Consequently, I budget at least half a day for any paper I am refereeing, and it often takes me longer. This is really what makes me put things off. Given this, excusing me from providing comments wouldn't save much time. I still have to be sure that I understand the twists and turns of the argument; I still have to ascertain whether the author anticipates my early worries when s/he gets deeper into the argument. Writing comments, especially short comments, is not a huge time add-on.

    As editor, however (of the Canadian JP — back in the late eighties, when response time was usually four months), I was the beneficiary of a system where I was allowed to reject any paper at any stage of the process. Back then, I would reject a good proportion of submissions before sending them on to referees. This didn't save me any personal time, since for most submissions a careful read was needed — but it did ensure a fast response to the author, since the paper failed to survive the first stage of the process.

    In short, then, it isn't comments or no-comments, but the possibility of exit at early stages of the vetting process that shortens average response time. At least, that's true given my habits and ways of reading.

    Reply
  7. Lani Roberts

    As a co-editor of a philosophy journal and the one who processes submissions for review, I would not want the submissions for comments only option because I have a difficult enough time getting good reviewers for the regular submissions, meaning the ones that are submitted for hoped-for publication. I am afraid if we included the "comments for revisions" option with our journal, the reviewers would get burned out and say "no" more often than happens now.

    Reply
  8. Richard Greene

    For the most part I echo Doug's sentiments. Still, however, I can imagine circumstances under which a "comments only for revision option" might make sense. For example, I had a paper a few years back that had been rejected a couple of times (in both cases I received good feedback from the referees). By the time I got around to submitting the paper for the third time (it went to a different journal each time), I had decided that I would not pursue that paper any further were it to be rejected, as I didn't want to devote any further energy to that particular topic. A "comments only for revision" option would have been great (assuming that it would really expedite the process, etc.). I suppose that persons nearing the end of the tenure clock or about to go on the job market who don't have a sufficient number of publications, might also benefit from this option.

    Reply
  9. Justin A. Capes

    In recent years I've gotten very quick "no comment" rejections from places like Nous and PPR only to have the same paper accepted at another top venue. While I appreciated the quick turn around times these places offer, as a young philosopher it would be helpful to know why the paper was rejected and how it can be made better. To be sure, a quick no comment rejection will make it possible to send the paper off to the next place faster, but if the paper is a non-starter or even if it has promise but needs a good deal of work, it would be nice to know this before sending it off to the next place. If the paper is no good or needs serious revision, it is a waste of the author's time and the time of the journal editor to send it out again. Even papers that are clear non-starters deserve some sort of report, even if it is just a brief note from the editor indicating that the paper is seriously deficient. If nothing else, such notes might prevent the same (unrevised) paper from gumming up the works at another journal. All this aside, would the "no comment" option really solve the problems with review time? After all, some journals do manage to provide detailed reports to the author in a timely manner, regardless of the decision.

    Reply
  10. Steven Hales

    Like Doug Portmore, I'm concerned that allowing a "comments only for revision" option will incentivize referees to reject outright, saving themselves the trouble of writing comments. If we could count on the professionalism of referees to not do so, then we could count on their professionalism to render prompt judgments. Which of course we can't, and that's the problem…
    Which raises this question, which I'll try to put as gently as possible: what the hell is wrong with you referees that take for-freaking-ever? I teach 4-4 and do all my own grading. I have a young child at home and a busy research program. Still I have never taken longer than three weeks to referee a paper, and I've reviewed for 19 different journals. Are the logjam referees seriously an order of magnitude busier than I am? Or are they just too important to bother?

    Reply
  11. Brian Weatherson

    Every time thus topic comes up, I'm shocked by how many people equate the time it takes to get a response from a journal with refereeing time. It does not take zero tome to

    * Process a paper for refereeing
    * Check that it should be refereed
    * Find a referee (this can often take many many months)
    * Review a referee report and make a decision on the basis of it

    I think the 'reject w/o comments' option would slow down many of these steps, and so probably shouldn't be universal.

    But the big point is that referees work much quicker than most commentators assume, when and if they finally agree to review papers.

    Reply
  12. Neil

    Brian Weatherson makes the obviously true claim that not all the time a paper is out at a journal is refereeing time. But it seems to me overwhelmingly likely that either almost all the time a paper is out at a journal is refereeing time or that the journal is not well-run. I edit a journal. I have an electronic processing system which works well. In most cases, I am able to make the judgment whether the paper should be refereed on the day of submission; with more complex papers or when I am pressed for time it might take as long as three days. Once I get referees' report I am usually able to make a decision as to final disposition on the same day. So the only step which is out of my hands and likely to slow the process down is finding referees. But this just *is* refereeing time (as Brian W reecognizes, in his epxlicit qualification to time spent on the paper *after* having agreed to do it). If the process takes more than a few days, it is because referees fail to get back to me with their decisions. So the delays are down to us, as referees, after all (or to badly run journals).

    Reply
  13. Mark van Roojen

    I agree almost completely with what Doug said, and I think Brian's comment above amplifies the point. If the problem at journals that take too long is not in the amount of time it takes to referee, then this idea would not actually solve the problems we have.

    I also think, that for journals that use more than one referee, this would make editorial decision-making harder, at least where referees disagree. It would be harder for the editor to decide how to treat the disagreement given the absence of reasons offered by one or the other referee. And if the editor's decision is to allow revising and resubmitting, how is the writer of the paper supposed to know what to do to please the silent but not happy referee? Does the editor go back to that referee for comments? Or does the editor now have to find a third referee to deal with the revised version?

    These threads are much better when we get some insights from actual editors about how journals work and what is creating bottlenecks or what parts of the process are the most difficult.

    Reply
  14. Chris Bertram

    Slightly different, but people may be interested in a paper by Bruno Frey and one of his co-researchers. It analyses some of the dysfunctional aspects of the revise and resubmit process and recommends a radical reform.

    http://www.iew.unizh.ch/wp/iewwp280.pdf

    Reply
  15. Enzyme

    I agree with CharlieH – a paper may warrant rejection, but still have salvageable parts (or it may simply be inappropriate for the journal chosen). I think that collegiality demands that reviewers give writers a fair insight into what they've done wrong – and, since it's presumably younger and less experienced scholars who're more likely to have papers rejected, there's a sense in which at least some feedback on rejected papers would be very desirable indeed as a means of encouraging and nurturing the next generation.

    Reply
  16. Thom Brooks

    I entirely agree with Brian Weatherson. I think many colleagues might be shocked at how much time is taken on all these steps. The 'reject without comments' would take more time and also (i suspect) lead to poorer quality comments: a worse possible world.

    Reply
  17. Steven Hales

    Sure, Brian, there are time delays in other aspects of the reviewing process besides just the referees. But I have been told by journal editors on *many* occasions that the reason I'm still waiting for a decision on my submitted paper (after 4, 6, 10 months) is that the editor is waiting on a delinquent referee.

    Reply
  18. Thom Brooks

    As a reminder, there will be an Association of Philosophy Journal Editors relaunch at this year's APA-Eastern in New York chaired by Carol Gould and I. It is scheduled for 28th December and will last two hours. Issues like the above will discussed. Fellow editors will be receiving personal invitations, but everyone (editor and non-editor alike) are most welcome for this session — which we hope will have strong attendance.

    Reply
  19. Brian Weatherson

    Steven,

    But you might only be waiting for a delinquint referee after n weeks because it took n-1 weeks to get someone to agree to referee the paper. That isn't compelling evidence that the referee is the problem.

    Neil,

    A lot of these steps should be sped up with electronic submission software. (They certainly made things dramatically better at Compass, and would help Phil Review a lot.) But the time to find a referee step (which you left out of your narrative!) is still an enormous burden.

    Reply
  20. Christopher Hitchcock

    I would like to echo Mark van Roojen's comment. This proposal would only work at a journal where standards or so high that a single referee's decision for outright rejection was sufficient for outright rejection. One illustration, many years ago I sent a paper to Synthese. There were two referees, one gave a very positive report and the other recommended rejection. The primary reason for recommending rejection was that the referee did not think the paper would be of interest to the readership of the journal. The editor sensibly asked around if people thought a paper on topic X would be of interest, and they said 'yes'. So the paper was accepted. This would not be possible if no reason for rejection was given. I have had similar things happen when I was a referee, or when I was consulted as a third party.

    Perhaps another alternative would be to give referees two deadlines: an earlier deadline for a decision (perhaps with a brief description of reasons), and then a later deadline for comments. I sometimes find that I have read a paper and know what I will say, but have to wait two weeks or so before I have the time to sit down and write up detailed comments. After receiving referees' decisions, the editor could inform the referees if comments would be needed to aid in decisions or for revisions, or whether they would just be for the benefit of the author.

    Reply
  21. John Turri

    I definitely believe Brian when he says that it can take a lot of time to find a ref. He would know, given his experience. But in my experience with journals that use "Editorial Manager" and "Scholar One" systems, it's the "under review" portion that takes the most time, by far. Journals using these systems include Synthese, Erkenntnis, Nous, PPR, Phil Studies and others.

    The stages seem to be:

    1. new submission
    2. editor assigned
    3. reviewers assigned
    4. under review
    5. reviews complete
    6. editor has a decision

    Again, this is just my own experience, but I keep close tabs on the progress, and almost invariably step 4 takes the longest, probably longer than all the other steps combined. Getting from 3 to 4 takes the second longest.

    Reply
  22. Tom Hurka

    As Mark van Roojen says, a referee is giving advice to an editor, who will make a decision based on that referee's report and perhaps one or two others'. And the advice isn't very helpful if it's just thumbs up or thumbs down. The editor wants to know what the referee's reason for the recommendation is, so he or she can judge whether it's persuasive. Some journals distinguish between comments for the editor, which give the referee's reasons, and comments for the author, which can be used in revision. And you could have an option where referees provide the first and not the second. But then they're still writing, and it's not hard to write comments that can also be sent to an author. So I don't see a great advantage here. And a no-comments/no-reasons refereeing system puts a lot of power in referees' hands and takes it out of editors' hands. As an editor (which like Mohan I was at CJP) I wouldn't want to get a pair of referees' reports, one saying simply yes and one simply no. That would be basically useless.

    Reply
  23. John Turri

    I meant to say 'getting from 2 to 3 takes the second longest'. I really don't know why they distinguish between 3 and 4!

    Reply
  24. Peter Brian Barry

    Tom Hurka's worry that an editor who received conflicting reports from referees–one approving for publication and one rejecting without comment–does suggest a significant problem. A dedicated editor could solicit further feedback from those referees, seek out (still more) referees or perhaps other measures that would surely take more time. But two questions:

    1) How do journal editors handle things now upon receiving conflicting reports? In my experience with a prestigious journal, when one referee recommended publication and one recommended rejection, the editor rejected the paper outright. Is the proposed "reject without comment" option more problematic than the present status quo?

    2) Suppose that authors who took the reject without comment option had to agree to the following policy: selecting the RWC option means that if any referee recommends rejection, the editor will reject the paper outright. In that case, the decision for editors is easy as well as justifiable; at least, authors would have no complaint even in the odd circumstance in which referee reports differ wildly. Would this policy be legitimate? Hurka's point that this puts a lot of power in the hands of referees is still a legitimate one, but it seems to me that referees already have a rather great deal of power in the present system.

    Reply
  25. Neil

    Brian W, I didn't leave it out of my narrative at all. And I don't know how it can take months to find referees, unless potential referees do not respond to invitations (which is why I claimed that the problem was due to the refereeing class). As John Turri notes, one can now often monitor these things, and this data suggests that it is time at referees that slows down the process.

    I have two new submissions in this morning. I will invite referees by midday. I expect a response within 24 hours normally, but the weekend will slow things down. In any case, in the actual normal course of events I expect to have referees assigned within 48 hours of receiving my submissions (I should add that my referees have been willing and efficient at all stages of the process).

    Reply
  26. Brian Weatherson

    I think I might have misinterpreted you Neil. Sorry about that.

    I didn't mean to say that the refereeing class us not to blame. People who don't reply to requests to referee, or reject without making good alternative suggestions, are very much part of the problem.

    All I did mean to say is that in some cases where an author has had to wait a long time, the writer of the review they get may have been part of the solution, not part of the problem. Put another way, there are a lot of people to blame when a review takes too long. That includes bad editors, something I've been guilty of, bad referees, and bad should-have-been referees. We won't solve the problems by just focussing on one of these.

    I suspect we agree on all this, so I'm sorry again for the misinterpretations.

    Reply
  27. Steven Hales

    John Turri's points about "Editorial Manager" and "Scholar One" are good ones. I've used those too, from both sides of the table. Surely those systems could be data-mined, and we would have some actual empirical evidence about where the bottlenecks are in the review process. We've all got loads of anecdotes, but perhaps this statistical data would help us (as a profession) to figure out promising solutions.

    Reply
  28. Andrew Moon

    John,
    Wait, is step 3 the time that the referees have the paper? Or is that step 4? Or is it at both steps?

    Reply
  29. Elizabeth

    It's worth noting that the policy of Economic Inquiry (discussed in the link that began this thread) is quite different from what's being discussed here.

    Rather than the opportunity to opt for "comments only if accepted," authors submitting to EI have the opportunity to opt for a policy of "no revise and resubmit." If their paper is accepted, they can choose to make further changes suggested by reviewers, but no such changes are required; if their paper is not yet in position to be accepted, it is rejected (with comments).

    This policy is not aimed at shortening the stage until a first decision is reached, it's aimed at shortening the (sometimes seemingly-endless) round of revisions after that point. I think it's a much more interesting idea than the "reject without comments" proposal.

    Reply
  30. Diego Machuca

    I couldn't agree more with Nikk Effingham. I myself, when I referee, feel the need to write down my reasons for rejecting a paper. We're in the business of philosophy, so we should be able to give reasons for rejecting a paper and to make a couple of suggestions to the author.

    Reply
  31. Tom Hurka

    Thanks to Elizabeth for pointing out the very different policy of Economic Inquiry, discussed in the link. That to me is a very* attractive policy. Philosophy too has gone to multiple rounds of revise-and-resubmit and I'm not convinced the result has been an overall improvement in published work. Yes, some papers plug holes pointed out by referees, but others add material demanded by referees that's of lesser interest, distracts from the paper's main point, and makes it longer. I think it's arguable that the net effect of revise-and-resubmit has been to make papers worse. A policy that lets authors eliminate that option, while still getting referees' comments, sounds very attractive to me.

    Reply
  32. matthew smith

    Take John Turri's timeline.

    The claim seems to be steps 2 and 3 take a lot of time. I wonder if it would take less time if editors farmed out refereeing to more people. That is, I wonder if there is too small a group of people to whom editors turn when looking for referees. If so, perhaps expanding the number of possible referees would be a good start.

    I've never edited a journal, so editors: can you illuminate us on the process of referee selection?

    Reply
  33. David Velleman

    An earlier comment asks for input from editors, so here's mine.

    1. By far — by very, very far — the largest contributor to the problem is the delinquent referee. We (Philosophers' Imprint) make decisions promptly upon receipt of referees' reports. Even if referees take the full amount of time that we allow (six to eight weeks), we can still give the author a timely decision. But as has been pointed out in other threads on this topic, a referee who misses our deadline — and then strings us along with broken promises — puts us in a very uncomfortable bind. Having already "invested" two months in one referee, do we start over with a replacement? Right now I have emails promising reports "next week" from two referees who were sent a paper in March. And those emails are already two weeks old. I will now have to send the author an abject apology and figure out what to do next.

    2. That said, I agree with Brian Weatherson that problems can arise at every stage of the process. And sometimes the fault does lie with the editor, since we occasionally drop the ball. When I drop the ball, the reason is usually that I have let a paper sit with delinquent referees too long. (A few of you out there have received abject apologies from me in such cases.)

    3. I don't think that simple thumbs-up/thumbs-down reports would be useful. But sometimes the reasons for rejecting a paper can be summarized in several sentences, and the problems pointed out in those summary comments swamp any other considerations. In some cases, we receive two or three single-spaced pages of detailed comments that don't clearly militate one way or the other, preceded by a summary that says "This paper just isn't sufficiently original" or "There is a crucial fallacy in the central argument" or something of the sort. In such cases, the detailed comments add nothing from the editor's point-of-view — though they may of course be useful to the author. And the question then becomes whether helping the author iron out minor problems was worth the time, which probably delayed referee's report.

    I am often amazed and even touched by the care that referees devote to commenting on a paper. But then I wonder: Suppose this referee hadn't planned to spend 4 or 5 hours on this paper no matter what the nature of its merits or flaws. Maybe the referee wouldn't have waited so long to read the paper — wouldn't have waited, that is, for a 4-5 hour window of free time before sitting down to the job.

    In other words, I suspect that a part of the problem is the way referees schedule their time. Responsible referees may put off reading a paper until they have enough time to write detailed comments — and then they write detailed comments, even if an editorial decision doesn't require them. (So it may turn out that the problem is due to an excess of responsibility on the part of some referees.)

    Christopher Hitchcock's suggestion is relevant here. I don't think that a referee can be asked for a quick decision. But maybe referees can be asked to give the paper a reading in a short amount of time and then to send a very brief report — either "This paper should be rejected, and here is my (brief) explanation why" or "I'll need more time with this paper; expect a fuller report in N weeks".

    4. Finding a referee can indeed take a fair amount of time. We sometimes have to send out 7 or 8 invitations in order to recruit 2 referees. Invitations do not usually get a quick response, and very few of them contain suggestions of alternative referees. So recruiting two referees can indeed hold up the process.

    5. On conflicting reports. We have no fixed rule: it all depends on the details of the case. Sometimes we will solicit a "tie-breaking" referee, if the disagreement between reviewers leads us to think that the tie-breaker really could go either way. Often we will reject the paper. Tie-breakers are indeed time-consuming, and as our volume of submissions increases, we will have to resort to them less often.

    6. "No revise and resubmit" already is our policy, in the sense that every paper is either accepted or rejected, although some rejections are accompanied by a note expressing our willingness to consider a revised version. Of course, authors can always choose not to resubmit. And, yes, sometimes the revision cycle ends with a rejection. I don't see any way of preventing that outcome, short of rejecting every paper that isn't perfect at the start. In any case "no revise and resubmit" has no effect on the problem. |

    Reply
  34. Doug Portmore

    Dear Professor Velleman,

    You write: "Even if referees take the full amount of time that we allow (six to eight weeks), we can still give the author a timely decision. But as has been pointed out in other threads on this topic, a referee who misses our deadline — and then strings us along with broken promises — puts us in a very uncomfortable bind. Having already 'invested' two months in one referee, do we start over with a replacement?"

    Why isn't the answer just obviously "yes"? The person obviously isn't trustworthy, having already broken his or her original commitment to submit a report within six to eight weeks. And what's the big "investment"? Of course, if you choose to seek out a replacement referee, you have go through the whole process of finding another person who is willing to referee the paper, which is, I realize, a real pain. But is it more painful or difficult than having to explain to the author the reason for the delay and to continue to nag the delinquent referee for a report? And isn't the worst that can happen is that you end up with three reports two months later? If editors automatically got replacements for referees who did not submit their reports within the promised eight weeks, then editors would almost always be able to render a verdict within four months. Instead, you're stuck with two referees who accepted the task in March and who continue to string you along with broken promises.

    Having dealt with you a couple of times myself as a potential contributor and having heard good things from others about your level of commitment, I have the utmost respect and gratitude for the excellent job that you've been doing. Philosophers' Imprint is, by all accounts, an excellent journal, which is extremely well run. But I just don't understand why editors don't automatically seek replacements for delinquent referees. In these threads, we're always hearing that the problem is delinquent referees, but I just don't see why the obvious solution isn't to seek replacements for delinquent referees. Not having experience with editing a journal, I must be missing something.

    Perhaps, the worry is that you may get the report one or two weeks after having secured a replacement. And it would be wrong to just tell that person "never mind." So you would have to wait for however long you gave that replacement to file his or her report. But I would think that it would not be unreasonable to ask referees to file their reports within six weeks. In that case, even if one of the original referees is delinquent, you can still expect to render a verdict after 12 weeks. I don't think too many of us would mind the risk that the process could be extended by six weeks if this greatly reduced the risk that it could be extended by 12+ weeks, as it has in the case of your two referees who were given papers in March.

    Reply
  35. Jamie Dreier

    Doug, if you get a replacement referee, you of course tell the original tardy ref they're being replaced.
    What makes the question difficult is that it may well be that with just a little prod the tardy ref will spring into action and finish the job much sooner than if you had to go get another ref.

    I'm an editor for ETHICS. I agree that by far the biggest time sink is an irresponsible referee, but I'll also second David's remark in (4): a paper I'm in charge of is now in review, but I had to ask 13 people to get two to agree. That's an extreme case, of course.

    Reply
  36. Doug Portmore

    Jamie,

    You write: "What makes the question difficult is that it may well be that with just a little prod the tardy ref will spring into action and finish the job much sooner than if you had to go get another ref."

    You're such an optimist. But, seriously, you should prod, giving him or her an extra week, and then replace him or her if no report comes in the next week.

    I realize, of course, that this could result in the process taking more time. But if you give the initial set of referees six weeks and then an additional week after that before replacing them, then even in those cases where one or more of the initial referees must be replaced, the process should still take no longer than four months. What I hear from editors and from various other people is that some referees take six months or more and that editors are still waiting on these referees. This is what leads to some of the outrageous response times that we hear about — response times exceeding 8 months. I estimate that it has taken 8 or more months on ten percent of my submissions to reach an initial verdict. And what I'm saying is although I realize that my proposed policy of replacing referees rather quickly would certainly increase response times in a number of cases, I think that it would greatly reduce the number of cases in which it takes more than six months to reach an initial decision. I think that many of us would be willing to risk that the response time might increase by an additional six weeks if this will greatly reduce the risk that it might take six or more months for an initial decision. There just doesn't seem to be any safeguards to prevent a referee from essentially hijacking a paper and letting it take six, eight, or more months — at least, not at certain journals that have been much discussed lately. I just want to understand why not have a policy such as the one that I'm suggesting that greatly reduces the risk of the initial decision taking more than six months.

    Reply
  37. David Velleman

    Hi, Doug (Portmore). There is the possibility that the tardy referee has already done some work on the paper and, as Jamie (Dreier) says, will come through with just a bit more prodding. And then there is the other problem, seconded by Jamie, that recruiting a replacement referee may take considerable time and effort. But you may be right: maybe we should give up on a referee as soon as he/she exceeds our deadline. Really, it's an empirical question: which policy would yield more timely decisions? It may be worth testing.

    Reply
  38. David Chalmers

    A modest proposal: The field should standardize on requesting referee reports within a month (or less). As many have pointed out, it doesn't take more work to referee a paper within a month than to referee it within two or three months. Yes, people are busy, and a few more potential referees might decline this way. But if it became standard practice (as it is in many fields) people would quickly get used to it, and the upside is much larger than the downside.

    While we're at it, we could also standardize on something like three days for a response to a refereeing invitation, and something like a week as an extension for delinquent referees (in both cases, after this period the editor moves on). Again, if this were standard practice, people would quickly get used to it. This way, it ought to be possible in principle for most journals to get average response time down to under three months.

    This practice would help even if pursued by individual journals, but it would work best if agreed upon by many journal editors collectively. I see that there is a journal editor meeting coming up at the Eastern APA. How about it?

    Reply
  39. Eddy Nahmias

    David Chalmers' suggestion is clearly the way to go. And I would add number 1 here:
    1. Time from arrival of submission until editors send it to two referees: 2 weeks
    2. Time for referees to respond whether they will referee: 3 business days
    3. Time for referee to submit report: 4 weeks (editor sends reminder at 3 weeks)
    4. Extension for referee without explanation: 1 week (with editor reminder)
    5. Extension with good explanation: 3 weeks MAXIMUM
    6. Time for referees to send to new referee(s) if needed: 2 weeks
    7. Time for editors to respond to submitter after referee reports returned: 2 weeks.

    I haven't edited a journal, so I don't know if these guidelines are realistic (esp. #7), but they seem so to me.

    The APA should take the lead in establishing such standards, and journals that don't adopt these standards should be rebuked in public (e.g., blogs) until they do or unless they have a clear reason to depart from them. And referees, knowing these clear and universal standards, should obey them or feel very guilty.

    Reply
  40. Eli Weber

    Here's an even more modest proposal, but also a bit harsher than what's being proposed by others. Hold delinquent referees accountable. If you don't meet your refereeing deadline for a journal, you can't publish anything in that journal. Even better, you can't publish anything in ANY journal. This could be a limited suspension, rather than permanent one, so that it doesn't destroy careers, and legitimate excuses provided in a timely manner could still be honored by journal editors. If the problem is with delinquent referees, maybe some consequences would help, because it doesn't appear that there are any right now. As a young philosopher, I'd like see something done, since my future career prospects are linked to my ability to publish articles in a timely manner.

    Reply
  41. L.A. Paul

    Eli:

    The immediate consequence of your proposal is that very few people will agree to referee papers.

    Reply
  42. Tony Lloyd

    Perhaps we should be looking to the economists.

    Does refereeing a paper "count" in reviews; tenure, salary, funding or otherwise? Does the referee gain any tangible benefit in reviewing a paper? Is that benefit lost if the referee is delinquent?

    Without the benefit it seems to me that an economically rational action would be to decline invitations to referee. If one, in a moment of irrational goodwill, agreed to referee a paper then it would be (economically) rational to work on your own paper before turning your attention to the review.

    If

    1. "I reviewed ten papers last year" gave value to the reviewer in excess of 40 to 50 hours spent doing something else and
    2. the reviewer knew he would be replaced if he didn't complete the review within David Chalmers' one month.

    Then the rational (in economist-speak) course would be to accept the invitation to review and to stick to the time-limits.

    Reply

Designed with WordPress