Leiter Reports: A Philosophy Blog

News and views about philosophy, the academic profession, academic freedom, intellectual culture, and other topics. The world’s most popular philosophy blog, since 2003.

  1. Necessarily Anon Faculty in a Red State's avatar

    I never met Alan, sadly, but his work leading the charge in making philosophy of sex “a thing” in contemporary…

  2. Paul Russell's avatar
  3. Facundo Bey's avatar
  4. Jane Byford's avatar
  5. Allen Hazen's avatar

    I taught in the philosophy department (“mental and moral sciences – philosophy”) at TCD in 1977-1979, and so had David…

  6. Brad's avatar
  7. An MA student's avatar

Guidelines for Journal Referees: Input and Ideas Sought

Brian Leiter

MOVING TO FRONT FROM SATURDAY (NOV. 27) TO ENCOURAGE MORE FEEDBACK

Thom Brooks (Newcastle) writes:

I was hoping that you could help publicize some comments for feedback from your readers concerning referee guidelines. I am part of an APA committee looking into journal practices and sit on a subcommittee on guidelines for referees. We have three different areas where I would be keen to solicit comments from your readers: 

A.    Appropriate response times to an invitation and for an evaluation

My view is that potential referees should have at least two weeks to respond to invitations for reviews. This is because they should have an extra few days to assess whether they have the time to commit to providing a review. Once a referee accepts, I believe the maximum evaluation time should be 6-8 weeks and ideally one month. Colleagues should have time to review given their many existing commitments, but no more than two months. Otherwise, it is best to find someone else.

B.    Guidelines for comments with regard to civility and the promotion of quality research.

Referees should be expected to treat authors with courtesy and even where authors have been uncharitable towards their targets. Reports should convince editors to follow their recommendations and uncivil reports may have the opposite effect.

C.    Conflicts of interest

While it is necessary that reviewers inform editors where they may know the author's identity, I am not sure this need always be a problem or that another reviewer is needed in every instance.

Thoughts from readers?

Leave a Reply

Your email address will not be published. Required fields are marked *

50 responses to “Guidelines for Journal Referees: Input and Ideas Sought”

  1. Peter Vallentyne

    I think that most journals can and should do a better job of processing submissions in a timely manner: They should contact invited referees after a week (not two weeks) if they have not heard back. Reports should be due in 4-6 weeks (not 6-8), with a request to aim for the low end. Obviously, there will be special cases with special circumstances, and they can be dealt with accordingly. The defaults (that should activate reminders, etc.) should be shorter than they are at many journals.

    Reply
  2. Stuart Elden

    Although not in a philosophy department, I have published in and reviewed for several philosophy journals, and edit a journal that has published work by philosophers.

    The second and third seem straight-forward: although especially with the second there are still plenty of uncivil reports. As an editor I don't ordinarily edit reports, but I might remove an unnecessarily nasty sentence or two. On the other hand, I do find it difficult when the report for authors is softened, perhaps suggesting substantial revisions, while the confidential comments to editors are much harder. The editor can then be in the difficult position of knowing the paper should be rejected but without the justifications to do so. So I'd encourage civility but directness. (For this reason I discourage comments just to editors, unless they are to inform instances in the third category, or other more technical issues.)

    The first is perhaps uncontroversial too: who could argue with giving people enough time to decide if they can commit to do something, and then giving them a sufficient period of time to do it? The time allowed is balanced by editor needs to keep things moving, and authors want to hear about their papers in reasonable amounts of time. But, in my experience as editor, this is what can introduce delays into the review system. Referees often take a long time to agree to do a report, and many do not reply at all, and then many miss deadlines, even if these were extended to accomodate their needs. Once a referee has failed to reply, or to deliver what was agreed, another referee is asked, and the process begins again. I'd happily give all referees two weeks to decide, and up to eight weeks to referee the paper, if we always heard within two weeks and then always received the report within eight. Ten weeks maximum time in review if all the first group of referees agree; twelve if you need to ask a second group to replace those who decline, etc. But I'm cautious about expanding the time if there is no guarantee of delivery. So this is the question about the 'no more than two months' maximum. Yes, if you know at the outset that it will appear in two months, no if that's when you are forced to give up on this particular referee and start again with another referee. In other words, this proposal only improves things if all referees reply within that time, and do what they agree to within that time.

    Reply
  3. John Schwenkler

    Regarding point #3, it seems to me very important to preserve the integrity of the blind review process, even in cases where knowledge of the author's identity won't leave the reviewer facing a "conflict of interest" in the ordinary sense of that term. The internet makes it extremely easy for a curious referee – and who isn't curious? – to determine an author's identity, and even when a referee who has such knowledge makes a concerted attempt to evaluate a paper strictly on its merits it's very easy for factors like race, gender, seniority, etc. to bias his or her impression of the work. Referees should be reminded of this possibility, and encouraged to avoid the temptation to uncover the author's identity until their reports have been sent in.

    Reply
  4. Lewis Powell

    At the beginning of this year, I started as the managing editor of the PPQ. I thought I might share some thoughts on how my practices relate to these suggestions.

    I rarely wait two weeks for someone to /respond/ to the request to be a referee before I start to seek other referees. Most people respond to referee requests in a few days, and some people don't respond at all (possibly because e-mails get missed or stuck in the spam filter?). At any rate, I have reminders set for myself to start looking for other referees if I don't hear back in four or five days. This practice has yet to cause any trouble with referees, but does accelerate the process of securing a referee, in most cases.

    We currently have a six week deadline for referees to return advice, though if an individual referee indicates that they would be willing to review the article, but that it will take them 10 weeks to return advice, for example, I typically conditionally assign them to review the manuscript, and continue with attempts to secure a referee who can meet the six-week deadline. I let the original reviewer know that this is my plan, and tell them that I will inform them if they are off the hook for refereeing. This practice has also not caused any issues with referees, but has helped bring down the time for review in at least a handful of cases.

    Most delays don't happen at the stage of agreeing to referee, however. The real issue is that six-weeks pass without the referee returning advice. On the assumption that at least some of this is due to forgetfulness (people often have lots of work on their plate, and it is easy for refereeing tasks to fall off the radar), I am working to set up automated reminders at the one-month mark, and right after the deadline, so that the refereeing work isn't forgotten.

    On point (C), I get the sense that I may be an outlier in how seriously I take the "blind" part of the blind-review process. Generally, the only way someone would wind up reviewing a paper where they know who wrote it is if they decide not to tell me that they know who the author is. The only situation in which it otherwise comes up is on revise & resubmit decisions, where the original referee learns the identity of the author after the initial evaluation before the revision is submitted. In that case, we have to balance the competing interests of having the same referee judge the revision versus preserving blind review. I typically defer to members of the editorial board in these rare cases.

    I am not aware of any issues with civility for referee reports at PPQ, but I've only been managing the journal for a short time. If anyone has (or has had) any issues with uncivil referee reports at PPQ (or, really, any other issues), I encourage them to contact me (ppq@college.usc.edu).

    Reply
  5. A grad who prefers anonymity

    I agree with John Schwenkler about the importance of the blind review process. But I don't think that the onus of maintaining its integrity rests solely on referees. Authors can also do their part. Authors can and should carry some of the responsibility, too. Authors who post drafts online might, once a paper is under review, take it offline. In preparing papers for review, authors can take special care not to make comments that would drop hints as to their institutional affiliation, educational history, etc. As much as possible, we can 'anonymize' drafts, e.g., specific names, to the extent that they immaterial to the paper (not citations to books or articles) could be changed to placeholders in review.

    Reply
  6. Anon post-doc

    Something I don't understand is why in several so-called "blind-review" journals, the editors know the identities of authors when deciding (i) which external reviewers to pick; and (ii) what decision to take in light of their reports. Wouldn't it be trivial for administrative staff to make these papers blind for the editors? Wouldn't this better exemplify the ideal of blind review?

    Reply
  7. Chris Tucker

    I think the review time should be 1 month, unless the paper is sufficiently long, say, over 9,000 words. If you give people more time than that, it just gives them more time to forget or acquire other pressing commitments. Plus, you are going to get people who are late no matter what you do. Better that they be late when the deadline is 1 month than two.

    Reply
  8. John Schwenkler

    The commenter at 11:34 is absolutely right: the task of preserving the integrity of blind review doesn't lie *entirely* with referees. But there are limits to what an author can do when, e.g., a talk bearing the same title as his or her paper was presented at a conference whose schedule is listed online. Similarly, the suggestion to take down online drafts is sensible, but Google has a pretty robust memory for this sort of thing, no? The key problem I see is with a lack of awareness of *why* blind review is important: if we all reminded ourselves more often of the possible effects of subtle bias and so on, then authors, referees, and editors (and their staffs) would all make more of an effort to keep it in check.

    Reply
  9. Jamie Dreier

    Anon post-doc;

    Yes, it's trivial, and most (? anyway many) journals are indeed double blind.

    But there is a cost. When an editor who is assigning referees doesn't know who wrote the paper, the chance of picking an inappropriate ref (the author's dissertation advisor, another member of her department, etc. — I once assigned a paper to its author to referee!) goes way up. Double blind journals just pay that cost (as do authors who submit to them).

    Reply
  10. V. Alan White

    With respect to the issue of returning referee comments in a timely manner, may I suggest that there be flexibility adjusted to the task at hand? I've returned some short papers in a day or two (especially if they were exceptionally good or bad), whereas other times on lengthy papers that were tough calls I've really needed over a month to mull things over. So maybe there should be a sliding-scale of review periods as length and difficulty demand (as judged by the editor), especially for journals that mix long pieces with briefer discussions and the like.

    I also agree that a week is plenty of time for response to an initial contact by the editor. Yes, some people take vacations sans email, but my guess is that not many do; in that case one should set up autoreply-out-of-office anyway just as a professional courtesy.

    Reply
  11. David Chalmers

    Regarding (1), I would urge standardizing on no more than a week for a response to the invitation and no more than a month for a review. These periods are already standard at many journals, and standards longer than this tend to lead to unacceptable delays (especially given occasional failures to respond or to review). Of course there can still be special cases, and an invitation letter can always say that if a reviewer will need more time (either for a response or a review), they should say so in advance. But on the face of it, adopting a two-week/two-month standard would do little to improve the current situation, which is agreed by most parties to need improving.

    Reply
  12. Eamonn McDonagh

    With regard to B, reviewers should avoid comments like this, (written by me based on many I’ve seen),

    “The level of English of your manuscript does not meet the journal’s required standard. There are a number of grammatical errors and many badly worded or constructed sentences. Please have the manuscript checked by a native speaker of English and refine the language carefully. Authors have the responsibility to present their papers in good English which can be understood by the journal’s readership without difficulty. Once this has been fixed, we can accept.”

    The reviewer wants the English improved, in some vague, general sense. There is no indication of what aspects of the English he wants improved (waffle about grammar notwithstanding), or of the existence of any specific errors. The author isn't given any real notion of where to start with regard to improving the English in their paper.

    The reviewer thinks that the ideal person to correct the unspecified language problems would be a native speaker of English. No other status, knowledge or qualification is specified.

    Scholars who receive a response like this should protest to the journal’s editor and demand that if their paper is to be rejected because the English isn’t good enough then they should be provided with an annotated sample of it that indicates the kind of errors that need to be corrected. Vague suggestions that the language needs to “refined” or "improved" should be rejected out of hand.

    Scholars who receive a response like this should be open about the reason their paper was rejected and encourage colleagues who have had similar experiences to protest as well.

    No, I’m not advocating that that language standards don’t matter. I’m just arguing that colleagues whose native language is not English not be treated like imbeciles and, where appropriate, that they be given specific, actionable indications of how the English of their paper needs to be improved.

    Reply
  13. Bryan Frances

    Will the recommendations have any teeth? Who is going to punish me if I don't referee on time? Who is going to punish journal X if they don't adopt the recommendations? Or is there a carrot instead of a stick?

    With a couple exceptions I have been a good and almost timely referee, but when I have been a bad person I don't think any recommendations would have made me more likely to do the right thing.

    Perhaps part of the recommendations will be 'Here is how to get people to adopt the following good practices:…' ?

    Reply
  14. Anon Jr Faculty

    Since time from submission to publication make such a big difference in the lives of junior faculty, it can be a minor disaster for the pre-tenured if the referee decides not to look again at a paper he or she has given a revise and resubmit to. If the paper goes out to new referees, this can add months and months to the process.

    This may not fall into the area of 'civility' but it seems like a matter of courtesy to commit to a second review if one gives out a revise and resubmit.

    It would also help authors on the clock to decide whether to revise and resubmit if journals were transparent about their process. Are they going to send it to new referees or to the same referees, for example? Authors should know this in advance. (Obviously, there are unplanned for contingencies but there could be some rule of thumb.)

    Reply
  15. Mark van Roojen

    The title says guidelines for referees, but it looks to me like what Thom is asking is about advisable journal policies. As a referee I make a good faith effort to comply with the deadlines the journal gives me. I don't think as a referee I have any obligation to do it faster than the editors request, but I think editors should want to give their authors a response within a reasonable time. And that speaks in favor of their setting some deadlines which referees should then follow.

    As editorial policies the guidelines seem very reasonable. I think one could reasonably give less than two weeks to accept or reject a refereeing request. That might mean that people on vacation miss a chance to reply, but as long as the journal isn't pissy about it, I don't see that this would offend anyone. I also pretty well agree that 6-8 weeks is the outside limit on what a journal should set as its regular policy for returning an accepted review (allowing exceptions for special circumstances usually negotiated in advance). A month would be reasonable as well, as Dave suggests, but I know I'd accept fewer refereeing requests if everyone had that schedule. (Maybe that's a reason I should welcome that standardization.)

    I would want to encourage editors to state the time frame in the request letter. Most do, but I've sometimes found myself looking for it and not finding it in the request letter. Relatedly, I appreciate reminder letters a week or so in advance of the due date provided they are written as reminder letters and not as "you missed a deadline" letters that just happen to be sent a week early (as one journal seems to send). Since many journals now automate "you're late" letters it ought to be no more trouble to automate a polite reminder ahead of time. For me that would have put me back on track the few times I've missed a deadline since I genuinely try to do what I've promised. Of course if a journal is not automated I don't think this should be a requirement because I recognize that this would take time to do one at a time.

    I think anonymous refereeing (by which I mean that the referee does not know the identity of the author) is worth striving for. So referees should convey to the editor that they know an author. I can see editorial discretion about whether this disqualifies a referee, and I also think that journals that use multiple referees per paper have less to worry about in this regard if it is a case where the other referee does not know the author. There are going to be trade-offs here, given the small number of philosophers, the distinctiveness of some people's writing, and also the fact that some papers develop the widely known views of their authors.

    I agree with a previous commenter that the duty to facilitate blind refereeing lies with all involved. Myself I don't think it requires keeping your papers off your website or elsewhere. That problem should be easily handled by referees controlling their curiosity. (Maybe getting PhilPapers updates with web publications could make this more problematic than I realize. Perhaps that speaks in favor of different titles for papers posted on line?) Authors should remove self-identifying information including examples that make clear what department you are affiliated with and "thank you" notes to named people. It is really easy to put '[omitted]' where these would normally go. I thought this was widely accepted practice, but recently I've seen too many manuscripts that make it too easy to know roughly who wrote a paper.

    FWIW.

    Reply
  16. A Different Anon Jr Faculty

    This is a bit off-topic as the thread has primarily to do with refereeing, but there has been some chatter of late in the blogosphere concerning the process of blind review and so I thought I would try to raise the issue here.

    Is there any mechanism in place for monitoring journals to ensure that the review process is blind and fair? [I realize that some journals do not promise double blind review, but some do and I wonder if there is anyone who can verify that a journal's review process is as blind and as fair as they promise?] If someone had concerns about a journal's editorial practice (not policies, but practice), who would she turn to? The editor?

    Reply
  17. David Wallace

    Two thoughts.

    (i) At least in my own field (philosophy of physics, so perhaps more insular than some) blind refereeing is a bit of a myth anyway. Realistically (and filtering out submissions that are obvious rejects) you can guess who the author is more than 50% of the time anyway. Indeed, in an era of online preprints, it's as likely as not that you've read the paper already. So by all means let's argue for not gratuitously violating the blindness principle, but we also need a refereeing process that can cope with the possibility that a given refereeing event isn't blind, and indeed with the possibility that blind refereeing isn't an option for a given paper.

    (Incidentally, I think the idea that we can somehow hide the identity of papers even though they've been posted online is unrealistic. The whole point of online archives is to make one's work public to as wide an audience as possible, and if the proposed referee isn't in that audience, something is badly wrong somewhere. Publishing online – whether on an archive or on your own website – is, in its own way, *publishing*. If you really don't want a referee to know that you wrote a given paper, don't put it on the internet.)

    (ii) I have mixed feelings about Eamonn McDonagh's point. I sympathise; on the other hand, I'm not clear that it's my job to educate people on correct use of English. (An author is of course at liberty to appeal if they think I'm just wrong in saying that the quality of their language is inadequate.)

    Reply
  18. Anonymous untenured

    Here's a suggestion to help ensure timely decisions: after three months without a decision, the author should be permitted to submit the paper to a second journal, which then has first rights to publish over the first journal. I suspect this would put due pressure on journals to hold reviewers to reasonable turn-around times, while also compensating authors for unreasonable turn-around times (by giving them another place to publish). As a young, untenured faculty who currently suffering from two separate nine-month wait times, this seems entirely reasonable to me.

    Reply
  19. Mike

    I'd second Dave Chalmer's line (the last time this topic came around),

    "A modest proposal: The field should standardize on requesting referee reports within a month (or less). As many have pointed out, it doesn't take more work to referee a paper within a month than to referee it within two or three months. Yes, people are busy, and a few more potential referees might decline this way. But if it became standard practice (as it is in many fields) people would quickly get used to it, and the upside is much larger than the downside".

    It's really not difficult to get a report in under a month. Most delays in reporting are just unnecessary.

    Reply
  20. Eamonn McDonagh

    @David Wallace, I'm not suggesting any philosopher set herself up as a writing coach. I am suggesting that if you think the English in a submitted paper isn't up to scratch then you give the author some idea why you think that.

    After all, you wouldn't respond to a paper sent to you for review by saying "This paper is not good philosophy, please resubmit with improved philosophy", and nothing more, would you?

    Reply
  21. David Sobel

    I agree with David Wallace that we need to be realistic in our hopes for (double) blind refereeing. This will involve realizing that, esp. in smaller subfields, often the papers most worth publishing will already be familiar to those best suited to referee the paper. In these cases double blind refereeing by the most suitable referees may be impossible. However, I agree with Mark van Roojen that we can and should expect referees to not positively try to find out who has authored the paper that they are refereeing (by, for example, searching online for the paper). Indeed, I recommend to journals to add the expectation that referees will not do so to their statement of what they expect from a referee.

    Reply
  22. Anonother anon postdoc

    I hope this isn't off-topic, but I would second the suggestions made above about the importance of taking all reasonable steps toward ensuring genuine double-blind reviews: for instance, papers' titles should be witheld from both editors and referees. As has been noted in other blogs, key-words would suffice. And editors should never know the identity of authors. That's not difficult to do, especially with the software used by many journals these days.

    One can't help wondering why a majority of journal editors don't take those simple steps. I suppose editorial work would be even more thankless if one reduced the power that comes with it. One also wonders why the profession doesn't stand up to this. Perhaps those with most power have the most to lose from genuinely blind peer review.

    Reply
  23. C. Sistare

    @Another Anon Post-Doc:

    I'm curious as to why/how omitting the title would icrease the 'blindness' of the review.

    Reply
  24. Jamie Dreier

    I also agree with David Wallace. I recognize (or have a strong suspicion about the authors of) almost half the papers I'm asked to referee. Blind is better, but it's hard to achieve. And also about correcting the English in a non-fluent paper. When a journal asks David Wallace to ref, it is drawing on his expertise as a philosopher of physics. An author who isn't fluent in the language of a journal can ask someone who is to help with the paper.

    Reply
  25. Eamonn McDonagh

    @Jamie Drier,

    curiously, this exact lack of expertise doesn't stop many referees from offering firm opinions and idiotic advice.

    Reply
  26. Steve Finlay

    A few comments about blindness: I agree that refereeing should be blind whenever possible, and I generally try to refuse to referee an article when I know the identity of an author, even where the editor doesn't care. But as others have pointed out here, blindness is becoming increasingly hard to achieve.

    I disagree with the suggestion that papers should be removed from personal websites when submitted. Referees have an obligation to suppress their curiosity while performing a review, and I agree with David Sobel's suggestion that this should be an explicit expectation. (If you really need to know who an author is, you can search after submitting your review). But online (pre-) publication is today an important part of scholarship, and it would be detrimental both to authors and to their audience if it had to be suspended for however months or years passed between first submission and acceptance.

    However, I think Another Anon Postdoc's suggestion of blinding paper titles has some merit. We're exposed to far more titles of papers than we are to the contents of papers–from conference programs, but especially now from daily PhilPapers emails. Since many of us now receive emailed lists of work in progress posted online in our fields, this might be the only way to retain some measure of blind review.

    Finally, I vote for a standard expectation of an initial decision to referee in one week, and a review in one month. It's not hard to do. People will sometimes miss the deadline whenever it is set. But we'll all benefit enormously from making this the norm.

    Reply
  27. C. Sistare

    However, I think Another Anon Postdoc's suggestion of blinding paper titles has some merit. We're exposed to far more titles of papers than we are to the contents of papers–from conference programs, but especially now from daily PhilPapers emails. Since many of us now receive emailed lists of work in progress posted online in our fields, this might be the only way to retain some measure of blind review.

    This makes sense. Thank you.

    Reply
  28. C. Sistare

    @Eamonn McDonagh:

    Now you seem to be complaining both that reviewers do not offer explicit advice on writing and that, when they do so, thier opinions are firm and idiotic.

    It is not clear to me how you think reviewers should respond to poorly written submissions.

    BL COMMENT: I think Mr. McDonagh's concern has now received ample attention, so I'm ending further discussion of it. Thanks.

    Reply
  29. Ben Blumson

    The focus on shortening the time taken to review papers seems to me misguided – what needs to be shortened is the time between the completion and publication of a paper. But shortening review times cannot shorten times between completion and publication, unless journals increase the number of articles they publish.

    If review times were shortened from two months to a month and the same number of papers received per month, the number of reviews received by journals per month would double. So if journals accepted the same number of papers per month, then their acceptance rates would halve.

    If acceptance rates halve, then papers have to be resubmitted more often before they are accepted, so the amount of time taken on average between the completion of a journal and its publication is not decreased by shorter review times.

    Shortening review times cannot shorten the time to the publication of an article, unless more articles are published.

    Reply
  30. A grad who prefers anonymity

    There is a simple solution to the problem of titles: Papers under review might have their titles replaced with "Journal Name, For Review #X."

    Reply
  31. Another anon postdoc

    Perhaps it could be useful to collate some of the issues on which there appears to be a consensus:

    – One week to decide whether to referee a paper;

    – One month to referee it;

    – Papers' titles should be blinded (certainly to referees, preferably to referees and editors).

    Those are recommendations for journals rather than for referees, but it would be good if the profession could take notice of them somehow.

    Reply
  32. Jonathan Ichikawa

    At least some journals adhere to their stated policies of using blind review only sometimes.

    It's one issue — an interesting one, worthy of discussion — just what the journal policies should be; but I'd also be interested to consider the extent to which journals do adhere to their stated policies.

    While we're on the subject of journal accountability, I think that the publication of various statistics by the journals themselves would be extremely valuable: the distribution of response times is the biggest one, but I'd also be curious to know about the acceptance rates and the time from acceptance to publication.

    This latter is, of course, something of a pipe dream. (In pessimistic moods, I could wonder about the former, too.) The fact is, journals just don't have much incentive to improve things for authors.

    Reply
  33. David Wallace

    Still on blind refereeing (and with apologies for Brian if this is derailing the topic):

    I think there's a danger of a kind of doublethink here (a danger that comes up a lot when material is put on the Internet, come to think of it). It's easy to think that somehow putting a paper on a personal website, or even on a preprint archive, is somehow a way of disseminating the paper among friends and colleagues without fully putting it into the flow of academic discussion. It isn't. If you publish something on the internet, you have *published it*, as surely as (and in many cases, more visibly than) if you'd published it in Mind. You have declared it to be in the public domain, and you should expect that interested academics in your field will come across it and look at it. (Indeed, isn't that why you published it online in the first place?)

    By doing that, in my view, you essentially waive the possibility of guaranteed blind review. If you don't want to waive that possibility, don't publish the article on the web before submitting it to a journal. Various people say that it's unreasonable to expect people in this day and age not to pre-publish on the web; I agree completely. Modus tollens applies: it's unreasonable to expect blind refereeing to be guaranteed, or (in some fields, and increasingly in all fields) even likely. Perhaps measures like exhorting referees to control their curiosity, or blinding titles, can help around the edges, but I have to say that they look to me like attempts to stop the tide coming in.

    (As a postscript, it's perhaps worth noting that physics made this transition a decade or more ago: *everything* in physics gets published on arxiv.org as a preprint, submitting an article to Physical Review normally just consists of sending them the preprint ID number, and blind refereeing disappeared long ago.)

    Reply
  34. Anonymous

    In all of the recent discussions of journal policies, no one (to my
    knowledge) has mentioned that two of the premier journals in philosophy
    are edited by the same person and published by the same company: Nous
    and PPR, both edited by Ernie Sosa and published by Wiley-Blackwell.
    Ernie Sosa is an excellent philosopher and, I am sure, an excellent editor. Still, this concentration of editorial control is worrisome, especially given
    the recent problems at both journals: as I understand
    it, both have had backlogs requiring a hiatus in submissions.

    Reply
  35. John Schwenkler

    I share Ben Blumson's desire for shorter time to publication, but don't follow his broader point. Assuming rates of acceptance stay constant, the only way journals will be dealing with more referee reports is if the shorter turnaround times for refereeing somehow lead philosophers to *submit more things*. The only exception to this would be the first few months, as reports prepared according the "old" policies for papers sent out several months back would be coming in at the same time as those prepared according to the "new" policies for papers received more recently. After that, there should be no difference at all, except in lag time.

    Reply
  36. Anon post-doc

    @ David Wallace. An author's ability not to put their stuff on the web addresses a worry about "being marked down" for having low status. However, my concern, and I suspect other junior people's, is about prominent or well-connected people getting "marked up."

    Reply
  37. N Williams

    The suggestion that paper titles be blinded for the purposes of review strikes me as a poor one.

    I can appreciate the desire to keep the reviewing process as blind as reasonable, but blinding titles falsely suggests that titles aren't parts of the papers under review, or at least that they aren't important parts of those papers. Given how frequently we decide whether or not to read a paper on the basis of title alone, titles are surely one of the most important elements of any paper. A paper's title conveys an enormous amount of information about the paper. In order to avoid the publication of papers with poorly chosen or inappropriate titles (or simply bad titles), they need to meet the reviewer's eyes as much as the rest of the paper.

    Reply
  38. Jamie Dreier

    Jonathan, ETHICS does publish most of the statistics you're talking about, in the annual Editorial. It might be better to have that info as a web page, I guess, but it's fairly easily available.

    Reply
  39. Lewis Powell

    Jonathan, EJP also publishes some of the statistics you mention: http://www.wiley.com/bw/submit.asp?ref=0966-8373&site=1

    As for PPQ, the older software makes collecting (and verifying) this information something of a pain, though I am in the process of transitioning the journal to newer/better software. I will hopefully have easily recoverable, reliable data of that sort which I can start publishing to the journal's website sometime in the next year.

    I am also working on getting a full description of PPQ's editorial practices online for authors to see, and have been working hard at getting a greater degree of transparency for authors in the meantime.

    On the issue of blind review, and in particular, title redaction: While I agree that redacting titles can (in many cases) help improve the blind review process (for the sorts of reasons indicated by Steve Finlay), it may have an unfortunate side effect. When I send out referee requests, I include the title of the paper and the paper's abstract in the body of the e-mail. I suspect some people get 'hooked' by the title, at least enough to read the abstract before deciding whether to agree to referee, and I wonder if there would be a trade-off in removing the titles between higher likelihood of a truly blind review in exchange for lower rate of positive responses from potential referees.

    To Anonymous (8:53am): For what it's worth, I think that a hiatus on accepting submissions in the face of a sizable backlog of submissions is responsible journal behavior, and not indicative of problems with the journal. Since submitting an article to a journal carries with it serious opportunity costs for authors (as almost every journal requires that articles under submission not be under submission at any other journals), it is important for journals to adopt policies which respect the costs that authors bear. Additionally, as a high volume of submissions places heavy burdens on the pool of (volunteer) referees used by the journal, it is important for journals to throttle submissions to ensure that their referees are not overburdened. This is all separate from the point you raise about whether there are issues with having a single editor/publishing house responsible for two of the most prestigious journals, but worth noting nonetheless.

    Reply
  40. David Wallace

    @Anon post-doc: I see that concern, but my point applies to it in spades: in the era of online pre-publishing, it is unavoidable that a well-known researcher in field X will have their pre-prints read or at least glanced at by most experts in field X, and so their referee (who obviously has to be an expert in field X) will know who they are. You can't achieve blind refereeing in this case, not without seriously distorting the choice of expert or closing down the pre-publishing framework altogether.

    (As a separate point, I'm not sure (and I'm really not sure, it's not just a rhetorical point) why junior people in particular should be worried about the possibility you're considering. "Prominent or well-connected people" presumably have tenure-track jobs already, so you're not competing with them. Most journals I review for don't seem to be especially constrained by overall page count (at most it delays the formal publication date if lots of papers are accepted) so publication is not a zero-sum game… but possibly that's not true of all journals, or possibly I'm missing something else here.)

    Reply
  41. Steve Finlay

    @ N Williams:

    I agree that titles are important, and that at some stage in the process we should want critical attention directed towards an author's choice of title. However, I can't accept that there is EVER a case where a title should be the determining factor in a review decision. (Although there could conceivably be cases where it is the determining factor in an editor's decision, as if the author stubbornly refuses to back down from a particularly stupid or offensive title).

    Reply
  42. Becko Copenhaver

    I like that several journals do a quick first round to determine whether to send the paper out for fuller review. This makes their job more efficient and ensures that those of us who do a lot of reviewing are not burdened with giving detailed comments to papers that would require a great deal of time and effort, perhaps with little benefit to the author.

    On the other hand, I get the sense that Mind, PPR and Phil Review do this in a way that leaves unanswered the principle reasons for rejection in the first round. I could be totally wrong about this and I am very happy to be corrected. I hope that this is just wild fantasy on my part.

    In my experience, authors simply hear that their papers won't be sent on for further review. I am decidedly not asking for detailed comments (this would undermine the whole point). But without one or two lines, the author is left with so little information that the mind is left to fill in with all manner of stuff that is probably inaccurate. For example, one may conclude that the paper is a hopeless failure (surely being rejected from these journals doesn't entail that). Or one may conclude that the first round is not blind – I assume they are. Or one may conclude, etc. etc.

    This is really just a matter of transparency. I know that PQ asks reviewers to put their response in one of five categories and asks for a few lines of justification. This seems reasonable and transparent.

    Reply
  43. Ben Blumson

    To reply to John Schwenkler, it is not that authors would respond to shorter review times by submitting more papers, but by submitting the same paper more often: if my paper were rejected in one month, instead of two, I'd submit it twice in two months instead of once. So submissions per month would increase.

    Reply
  44. Thom Brooks

    Dear friends and colleagues,

    I am absolutely delighted that this thread has opened up so many highly useful comments. In my brief statements on A-C above, I only meant to indicate the boundaries for acceptable reviewing and I would share what appears to be a consensus view that I was too generous. Readers may know I'm editor and founder of the Journal of Moral Philosophy. The JMP waits about 1 week to secure referees, we ask referees to provide reviews within a month, and, in turn, we're able to review about all papers in 6-8 weeks. (We also have an 8% acceptance rate, for what it's worth.) So I think that many journals — I don't think the JMP unique at all — satisfy many of the deamnds readers have stated. Nevertheless, a few comments on the comments. There seems to be a consensus on journals:

    1. asking referees for reports in no more than 4-6 weeks.

    2. offering referees no more than one week to decide whether or not they should review.

    3. taking appropriate measures to guarantee anonymity (with responsibility shared with authors, who might be encouraged to remove online papers during a review). I entirely accept Jamie's note on the potential costs which would ensure certain measures to avoid problems of the kind he mentions.

    4. should ensure that authors remove the identities of persons named in the acknowledgement notes (although it would be helpful to provide full disclosure to the journal).

    5. should publish statistics on submissions. (The JMP publishes its information on The Brooks Blog.)

    Some considerations for further comments:

    A. Would readers support the norm that reports need not be provided where decisions to reject are determined in four weeks or less (although reports are always desirable)? This might help speed up reviews at the cost of less feedback.

    B. Would readers support the norm that journals provide a hard copy of the issue their work appears in or is a PDF file satisfactory? I know many publishers are moving away from providing hard copies.

    C. Some journals ask authors to recommend potential referees. Would readers support this norm or does it have too many costs?

    D. Many journals have a not insignificant "desk rejection" rate where submissions are rejected prior to review. The benefits are swift response rates and it saves editors the time for reviews. Few journals (to my knowledge) run full reviews with reports for all submissions. What do readers think?

    E. Some journals are thought higher in the "pecking order" than others. Should referees apply different standards in reviewing submissions for journals depending upon that journal's general reputation?

    P.S. Readers may be interested to know there will be an Association of Philosophy Journal Editors roundtable panel at the APA-Eastern on the topic of electronic publishing. Panel members include Julia Driver, Gary Gutting, Jeff Dean (Wiley-Blackwell), and yours truly. All are welcome.

    Reply
  45. Lewis Powell

    @ Thom:

    Your item (A) might be preferable to a lot of authors if it is refined in light of Becko's comment. Perhaps one thing that could come out of the APA committee's recommendation is a small set of reasons for quick rejections so that these fast-turnaround rejections can simply choose options like "reject: focus too narrow" or "reject: does not appear to advance debate" or something like that (including an option of "reject: other (please specify in one to two sentences)". If this list was standardized fairly well, it would be a boon for authors, who could get a good sense of how their paper is perceived, so that, if they receive one of these rejections, they would know "I need to make it clear in the abstract/introduction that this point generalizes" or "I need to make it clear how my solution differs from so-and-so's", etc.

    One major point that was raised, which didn't make your list, is one of process transparency. Authors should know how the evaluation process differs from journal to journal when deciding where to submit their work. Publishing statistics is good also (perhaps more important), but there is no reason why authors should be in the dark about the general "flow-chart" for the evaluation process at a journal.

    The reason I think transparency is so important is that unappealing/bad journal practices aren't nearly as problematic for authors if the authors know which journals observe them.

    One other issue that I think is worth mentioning: In general, all journals should move to a purely electronic submission system. Some journals still appear to require that authors mail multiple copies of a manuscript in order to submit it, and I think we should transition away from that.

    Reply
  46. Bill Wringe

    I'm late to this thread, but I'm going to mention something which no-one has brought up, which is clearly relevant to the topic, and which I at least think is important.

    I recently received a decision from a fairly high-powered British-based journal, which gave me a rejection, with comments, but for which the editor specified that the comments were provided on the presumption that I would treat them as confidential. it was my impression that this was a form letter, so that it represented
    a general policy.

    It was the first time that I'd submitted to the journal in question, so I don't know whether it has been standard practice for that journal for a long time. It also seems to be something of an outlier: in the past 10 years, I've submitted papers to – I guess – most of the top 10 journals in the field and I've never previously encountered a request like this before.

    I bring this up because it seems both unjustified, and pernicious. It's pernicious because it means that bad refereeing – and I hate to say this, but there is some bad refereeing – by which I mean, refereeing which is by any standards shoddy, ill-informed and frankly unprofessional, and not simply refereeing which fails to recognise the manifest brilliance of my own philosophical work 🙂 – out there – doesn't come to light; and it's unjustified because if the reviewers comments are blind, it is hard to see why they need the extra protection of confidentiality. I'm also inclined to suspect – though others may be better placed to confirm or deny – that allowing referee's reports to be confidential as well as blind drives down the care and quality of thought that goes into them.

    So I'd suggest that we make it a norm of the profession that, at least when they are blind, referees reports should NOT be confidential. It's good for the quality of refereeing and its good for transparency.

    Reply
  47. Bill Wringe

    Actually, I don't see any reason other than the self-protection why I shouldn't make public the name of the journal which does not want authors to be able to discuss the quality of the referees reports it is relying on.

    But self -protection is a fairly significant consideration here. The journal in question is REDACTED FOR REASONS OF PROFESSIONAL SELF_PROTECTION.

    Reply
  48. John Schwenkler

    @ Ben (if I may): I still don't think the problem you identify would persist beyond a few months of backlog, as the welcome side-effect of quicker resubmission would be a paper's spending less time "on the market", as it were. If the review process were more efficient, then a paper that right now takes (say) eighteen months to go through two rounds of rejection and then an acceptance would instead take (say) six or twelve months to go through those same stages and reach its endpoint, after which it wouldn't be a bother to anyone who didn't choose to read it in its published form. That paper you submit twice in two months instead of once would now be a done deal; thus unless standards change, you'll be submitting it the very same number of times overall.

    Reply
  49. Mark van Roojen

    Thom,

    On C, I think that allowing authors to suggest reviewers is just ripe for abuses and would make things even less blind. I know that some presses allow referee suggestions for book manuscripts, but book reviewing is also not blind. Since referees commit more time to a book manuscript than to an article there may be good reasons for the difference. But whether this practice is a good one for books or not, I think that as blind as possible reviewing for journal articles is worth striving for, even if we cannot achieve perfect anonymity.

    Reply
  50. Matt Shockey

    I agree with Lewis Powell regarding A, and would say the same applies to D.

    Regarding B, hard copies look nice on my shelf, and it gives me a warm, fuzzy feeling when I get one in the mail, but they serve no real function beyond that.

    No view on C.

    Regarding E, different standards depending on perceived place in the pecking order seems reasonable. There are things worth publishing that, for whatever reason, aren't good enough for the best journals. Publication of acceptance rates would help here too. That would help guide both reviewers and authors in their expectations.

    One other thought on the topic of revise-and-resubmits, which has been touched on but not much discussed: it seems reasonable to expect that in agreeing to review a paper, the reviewer is thereby committing herself (barring unusual circumstances) to reading a second version, if she recommends a r&r the first time. Otherwise, what's the point? A different reviewer might think the first draft is fine, or that an entirely different set of changes is needed, leading the author to waste lots of time trying to please different tastes.

    Reply

Designed with WordPress