Leiter Reports: A Philosophy Blog

News and views about philosophy, the academic profession, academic freedom, intellectual culture, and other topics. The world’s most popular philosophy blog, since 2003.

  1. Fool's avatar
  2. Santa Monica's avatar
  3. Charles Bakker's avatar
  4. Matty Silverstein's avatar
  5. Jason's avatar
  6. Nathan Meyvis's avatar
  7. Stefan Sciaraffa's avatar

    The McMaster Department of Philosophy has now put together the following notice commemorating Barry: Barry Allen: A Philosophical Life Barry…

A Proposal on Refereeing

Brian Leiter

David Chalmers (ANU) makes the following proposal in the comments to the earlier thread:

The field should standardize on requesting referee reports within a month (or less). As many have pointed out, it doesn't take more work to referee a paper within a month than to referee it within two or three months. Yes, people are busy, and a few more potential referees might decline this way. But if it became standard practice (as it is in many fields) people would quickly get used to it, and the upside is much larger than the downside.

While we're at it, we could also standardize on something like three days for a response to a refereeing invitation, and something like a week as an extension for delinquent referees (in both cases, after this period the editor moves on). Again, if this were standard practice, people would quickly get used to it. This way, it ought to be possible in principle for most journals to get average response time down to under three months.

This practice would help even if pursued by individual journals, but it would work best if agreed upon by many journal editors collectively. I see that there is a journal editor meeting coming up at the Eastern APA. How about it?

So how about it?

Leave a Reply to dankaufman! Cancel reply

Your email address will not be published. Required fields are marked *

28 responses to “A Proposal on Refereeing”

  1. Christopher Hitchcock

    I think it would be great if the APA adopted official guidelines like this, and if the association of journal editors got on board. I suggest that similar motions be put before the appropriate professional organizations in the U.K., Canada and Australasia as well.

    Reply
  2. Doug Portmore

    Not surprisingly (given my comments on the other thread), I'm all for it.

    As a potential contributor, I endorse it, because it would, I suspect, reduce the average response time and would certainly minimize the now too common phenomenon of a journal's taking over six months to reach an initial decision because one the referees has been grossly delinquent in his or her duties.

    As a referee, I'm all for it. After all, it is rarely more difficult to file a report in four weeks than to file one in eight or more weeks. About half the time, I procrastinate and then do it a week before it's due. I always try to do it at least a week before it's due so that even if something unexpected comes up, I'll still be able to meet the deadline. Moreover, I think that this policy would increase the chances that I'll agree to any request to referee. I generally decline only if I already have a number of other papers that I've agreed to referee and still need to file reports on. Since this policy shortens the time that I have to file my report, it makes it less likely that I'll get hit with a request to referee while I'm already sitting on two or more papers that I need to referee.

    Reply
  3. Eric Chwang

    I'm still a newbie, but I question D. Chalmers's math. I've submitted to *Bioethics* several times, with a median response lag of six months, yet when I referee for them my deadline is three weeks.

    Which reminds me–better get back to that referee report…

    Reply
  4. John Turri

    It would be fantastic if David Chalmers's suggestion were adopted. And Christopher Hitchcock's suggestion would make it doubly- or triply fantastic.

    Reply
  5. Fritz Allhoff

    Great; three cheers for Dave!

    Reply
  6. Peter Kail

    Writing – as many of us are – as someone from both sides of the process, David Chalmer's proposal is quite excellent. It is simply far too easy when you have a long deadline to treat such reports as having a low priority, which gets pushed lower and lower as one is asked to do other things in the intervening period. So bravo to this proposal!

    Reply
  7. Steven Hales

    I'll also climb on the bandwagon here. The key to making it work is standardization–if it were to become a profession-wide norm, then authors and referees both know what to expect from the system. One of the worst things in submitting a paper isn't rejection, it's having no idea if you'll get a decision in one month or one year. Philosophers are a wayward crew (we don't have a standard citation format, for example, as our cousins the psychologists do), but hopefully we can see that this is in our collective self-interest.

    Reply
  8. Paul

    I think it is probably unlikely that anyone who is taking 9 months to referee a paper is doing anything other than making the task a low priority. Adopting this suggestion would be a good way to change that, and I'm all for it.

    Reply
  9. Sandy Goldberg

    I also am in favor of Dave's suggestion. The editorial policy he suggests is not that far removed from what I take to be already in place at Nous and PPR, and they have hardly suffered from imposing this policy. Quite the contrary: as people are increasingly aware of how well-run they are, they have been swamped with submissions. (And while we're proposing things that might profitably be discussed by the journal editors that will be meeting at the APA: perhaps those journals that have not yet moved to automated submissions might be encouraged to do so. I confess, though, that I don't know the details of e.g. cost and such, which might be what is holding back those that have not already taken the plunge.)

    Reply
  10. dankaufman!

    yes, please.

    (eric, get back to work!)

    Reply
  11. Steve

    Just a quick response to "Yes, people are busy, and a few more potential referees might decline this way. But….":

    Folks at research-intensive institutions who have cushy teaching loads are, of course, nevertheless very busy; however, their very busy schedules are arguably much more flexible than those of folks with heavy teaching loads–some of whom, as Jason Stanley noted on this blog some years ago, still manage to develop "impressive records of publication," and who make key contributions to the areas in which they work.

    My guess is that, very probably, it would be the folks at these non-elite institutions who would have to decline (or who would be less likely to be invited in the first place, regardless of their qualifications) if invitations as a rule set "within a month" deadlines for referee reports. (I can referee a paper within a month or less during the summer months or during a sabbatical, but not during a typical semester, when I teach 3-4 classes 5 days/week to 80+ students with no graders or teaching assistants.)

    So one upshot of the proposal, I suspect, would be a forced decline in participation in a central professional activity for those who don't have relatively cushy teaching (& service) loads. Probably, the pool of philosophers very actively involved in a central professional activity would shrink and grow less diverse. I'm not sure how important that diversity is, but I am confident I'd be regularly disappointed at having to turn down invitations to referee that happen not to come in within one month of a summer or winter break.

    Reply
  12. David Velleman

    Thanks to Dave Chalmers for this suggestion. Before climbing on the bandwagon, I'd like to think it through.

    1. I would not prefer a 1-month deadline to alternative policies that have been proposed. We (at Philosophers' Imprint) often have strong preferences among possible referees, based on considerations of expertise, independence from the author, and so on. The shorter the time allowed for refereeing, the less likely it is that one of our preferred referees would feel able to finish the job in time, and the result would be more refusals from the best candidates. So I would prefer to allow a longer period, especially for papers that require close study and commentary.

    2. Still, I might prefer a general agreement to the current mess, if the agreement actually changed habits in the profession. And if such an agreement were for a deadline of N months, then I might prefer an N-month deadline simply because it contributed to, and benefited from, the development of better professional habits.

    3. But the considerations in (2) do not support a deadline of N = 1 month rather than N = 6 or 8 weeks. And, as I have suggested in (1), a few additional weeks may increase the probability of recruiting the best referees.

    4. Against the considerations in (2), I would weigh the significant differences among journals with respect to the kind of work they publish, their policies for refereeing, their resources for managing the editorial process, and so on. I venture to suggest that scientific journals are more alike than philosophy journals. The scientific research paper follows a standard format, whereas philosophical research can fall into many different genres — mathematical, historical, literary, social-scientific, and so on. These genres require very different kinds of refereeing. (Also, I would want to resist too much standardization among philosophy journals, but that's topic for another thread…)

    5. All things considered, I would prefer a different standard from the one that Dave suggests. It would be a second-order standard — namely, the standard of meeting whatever standard is set by the journal for which one agrees to referee. If people took their refereeing commitments seriously — as seriously as they take other commitments that bear on someone else's career — then the problem would go away. Almost everyone writes letters of recommendation in time for job-application deadlines, because they know that someone's career is at stake. Well, someone's career is often at stake when a paper is being refereed. The editor's deadline is not primarily for the editor's benefit; it's for the benefit of the author. If everyone kept that in mind, the process would go smoothly.

    6. Finally, I think that journal editors must be constantly on the alert for ways to improve their own editorial processes. These discussions are extremely helpful to those of us who are trying to oil the wheels of academic publishing.

    Reply
  13. Joshua Earlenbaugh

    "If people took their refereeing commitments seriously — as seriously as they take other commitments that bear on someone else's career — then the problem would go away. Almost everyone writes letters of recommendation in time for job-application deadlines, because they know that someone's career is at stake. Well, someone's career is often at stake when a paper is being refereed."

    yep…

    Reply
  14. Ofra Magidor

    I like the suggestion to standardise report times, but let me add two points:

    1. Like a few others above I worry that 1 month is a bit too short. If instead we go for 2 or 3 months that would mean that any request is likely to include within its deadline a term break, which on the one hand will make things much easier on referees with heavy teaching loads, but on the other hand would still be a huge improvement on the current review times. Sure, writing a review doesn’t take more than a day or so, but finding a full day of work during a busy teaching term may not be all that easy.

    2. I would like to hear a bit more about how the suggestion to standardise report times is supposed to be enforced. One option is to agree that referees are simply automatically unassigned to the paper if their report is not submitted by the deadline (this would be especially easy to implement in journals with electronic submission systems). Of course this runs the risk of increasing review times rather than decreasing them (because some referees might waste 2 months, only to have the paper pulled out and reassigned to another referee). Still, I suspect that if referees knew the deadline is really rigid most will turn in their reports on time. At any rate, I doubt that merely deciding in principle that one ought to return reports within N weeks will be enough to change the current situation – so I think we need to put in a bit more thought into how to ensure these standards are put to practice.

    Reply
  15. Stephen Nayak-Young

    Steve and David Velleman raise significant worries, namely, that busy teachers may not be able to meet the 1-month deadline during the school year and that preferred referees won't accept refereeing requests.

    What about modifying the proposal such that every journal says to every prospective referee, "Our standard deadline is one month," with the understanding that it is negotiable, though not so negotiable as to be lightly doubled or tripled. Such flexibility can be abused, but it need not be, especially if any extended deadlines are negotiated upfront and then treated as "set in stone."

    One worry about this modified proposal would be that the authors would no longer have the same certainty about how long the refereeing would take (although, of course, they wouldn't know how long it took the editors to find a referee even if the 1-month deadline were inflexibly standard). I'm not sure how to handle this worry, but at least the knowledge that a one-month deadline is standard would give authors a better "ballpark" than "it could be six months, maybe a year, then maybe you could e-mail, etc."

    Reply
  16. Blake

    Brian,

    A few months ago I sent you a table from a major economics journal that listed how many articles were submitted, how many rejected/accepted/needed revision, and how long all this took(from submissions to review-review to publication etc.) I wonder if including such a table in a journal as standard would at the bare minimum illuminate the schedule on which a journal tends to operate. Obviously this data could be manipulated, but I imagine we'd all like to think we're above that-particularly at the pointy end of the journal world. Having worked at one of the world's biggest academic publishers I can tell you first hand that this information is captured in many other disciplines, if not always put in the journal.

    Reply
  17. Michael Weisberg

    Following up on Stephen Nayak-Young's response to David Velleman … I think a very reasonable option is for the action editor to say something like the following to the referee:

    "We try to respond to authors within 6 weeks, and hence would appreciate hearing from you in one month. However, we value your input, so if you think you will not be able to give us your opinion in that time, please propose an alternative date as soon as possible."

    I am not sure that the wording is exactly right, but the idea is to set up that second order standard in the referee and also nudge the time horizon down. At the same time, if this referee is the optimal one, everyone will have a clear understanding of the length of time involved.

    Let me also say that the PhilSci journals are starting to adopt the 6 week turn around, 4 week referee norm. BJPS has been a big leader in this area … and they make it work by being completely upfront with everyone about their expectations of timeframe. The other thing that they do is to remind the referee about the upcoming deadline with one week to go and after it has passed. They are also excellent about communicating with authors about the reason that the 6 week deadline has passed when it does pass. (And in my experience — they keep it at 2 months max.)

    Reply
  18. Thom Brooks

    I am in general agreement with Chalmers's proposals, although I share some of Velleman's reservations. I'll have this added to the Association of Philosophy Journal Editors agenda at the APA-Eastern.

    Reply
  19. David Chalmers

    Re standardizing on a period longer than a month: realistically, this would not be a reform. Most journals now work with a requested period in the 1-2 month range. I think that a reform that makes a significant difference would require (i) a 1-month deadline and (ii) enforcing this deadline reasonably firmly, in part by the other suggested measures. So I think that in effect, the choice is between something like this reform and no reform.

    Re Ofra's concern about enforcement mechanisms: the idea is that after exactly one month, an email goes to referees noting that the report is overdue and giving a hard deadline of one week. If the referee says they can't do it within a week, the editor moves on to a new referee immediately. Otherwise the editor waits a week, and then if no report is received, moves on right away. Of course this is easiest with an automated system but ought also to be possible for a journal with regular editorial assistance.

    I note that this system works much better with a 1-month deadline than with a 2-month deadline. With the longer deadline, the editors' dilemma (described vividly by David in the other thread) of whether to wait or move on arises much more acutely. (2 months of sunk costs, 2 more months of waiting if one moves on.) With a 1-month deadline, it's much easier to simply move on. And when a referee fails to come through, the resulting turnaround time is still humane.

    Re David's reasonable concern that this would make it harder to get good referees: My suspicion is that if the practice became standard, people would get used to it fast. But of course it is an empirical question.

    Re Steve's concerns about diversity: Again, the point is well-taken. At the same time it is not clear that it is reasonable to ask an author to wait a couple of extra months just to increase the diversity of the referee pool.

    In any case, Steve's concern, and David's to some extent, could be dealt with by implementing another proposal I have thought about. It has been noted that (i) one of the hardest part of a journal editor's job is finding appropriate referees, and (ii) there are many philosophers, especially junior philosophers, who are willing to referee more papers but are not on editors' radar screens. What is needed is a mechanism for connection — most obviously, a sort of referee database.

    David Bourget and I, at the PhilPapers project, are willing to implement such a database if there is a sense that it would be worthwhile. Anyone with a philosophy Ph.D. would be encouraged to make an entry in the database listing areas of refereeing competence (by canonical area and by keyword). Journal editors would have access to the list and could search by area, by keyword, and so on. Results would be linked to PhilPapers profiles with lists of publications. Potential referees could also add a comment with relevant information for editors, visible whenever their name appears in the results of a search.

    The problem for potential referees in Steve's position could be dealt with by the use of a comment such as "I am most able to accept referee requests in months X, Y, and Z." The problem for editors in David's position would at least be mitigated by having more information about expert referees. Of course it's an open question to what extent really senior philosophers would sign up for this system (just as it's an open question to what extent journal editors would use it). But at the least, I would expect that many very competent junior and mid-level philosophers would make entries in such a database, expanding the pool of expert referees that editors know about and can call on.

    I'd be interested to hear others' thoughts on the desirability of such a database.

    On the 1-month reform, I have emailed some journal editors and have had some positive responses. Of course some are doing close to this already, as Michael Weisberg notes. But having public commitments from a number of journals will help encourage others. Once there is some sort of critical mass here, I will report.

    Reply
  20. Jonathan Livengood

    For what it's worth, we don't appear to be as badly off as statisticians …
    http://www.biometrics.tibs.org/carroll.pdf
    (This paper is a bit dated now, but I have it on good authority that not much has changed in the last decade.)

    Reply
  21. Jeff Horty

    OK, this officially counts as work-avoidance, but someone asked for statistics. We (at the JPL…) ask reviewers to get us reports within six weeks. I am new to journal editing and have been surprised and delighted at how good and responsible our reviewers have been – this does not mean there aren't a few horror stories.

    Anyway, so the JPL uses the EM system, which generates statistics pretty easily, so I looked at the "Reviewer Performance" section of the "Journal Accountability" report, reproduced below – sorry if formatting is screwy.

    Six weeks is 42 days, and our reviewers as a whole seem to be getting us reports within 41.3 days of accepting an assignment, or 46.9 days of being asked (it takes them 5.5
    days to accept, yes there is a rounding error).

    The way the EM system works, a robot starts to nag reviewers within two weeks of deadline, then nags them more insistently once the deadline passes. If too much time goes by, the handling editor will generally lend a human touch to the nagging.

    Our total turnaround time till first decision appears to be 70.8 days, for the roughly one-year period I surveyed. Where do the other 23.9 days go? Finding a reviewer who will say yes. Reading the report, comparing it to the paper, thinking. Sometimes going back and forth between author and reviewer to clear up confusions before reaching a decision. Writing a letter to the author in non-straightforward cases. (Oh, and sometimes editors have other responsibilities too…)

    So those are some statistics. My opinion, or instinct, on the one-month suggestion is that it's too short, that too many referees will turn down an assignment with a one-month deadline. So it could be much harder to find referees. This could actually wind up slowing down the whole process. But if someone else wants to perform this experiment, I'd like to know the result.

    Also, I don't know too many tardy referees who wouldn't be just delighted to be punished by having their assignment taken away from them! What to do with a tardy referee is really a matter of individual judgment, and it helps to know the people. Many responsible people just get busy, sick, overcommitted. There are a very few irresponsible people, I've noticed. Sometimes you have to pull the plug. But I often feel, if a referee's been sitting on a paper for 2.5 months, say, they've got some really excellent guilt going, and we can use that guilt to extract a report.

    ***************************************

    Reviewer Performance Averages

    This section includes some key statistics about the
    peer review process. Unless otherwise specified, all
    calculations are based on reviews completed during the
    time period.

    * Days to Respond to Invitation

    Average number of days between date Reviewer was
    invited and date Reviewer agreed or declined to
    review.

    5.5

    * Days to Complete Review (from Date Invited)

    Average days between date Reviewer was invited to
    review and the date the review was completed.

    46.9

    * Days to Complete Review (from Date Agreed to Review)

    Average days between date Reviewer agreed to the review
    invitation and the date the review was completed.

    41.3

    * Number of Reviews per Reviewer

    Average number of reviews completed by each Reviewer
    during the time period.

    1.2

    * Average Days Late

    For Late Reviews, the average number of days those
    reviews were submitted after the due date.

    30

    Average Days Early

    For Early Reviews, the average number of days
    those reviews were submitted on or before the due
    date.

    33.7

    Reply
  22. Richard

    David Chalmers's suggestion of a database for willing reviewers strikes me as an excellent idea – not least because it might also be a good way for willing reviewers who lack contacts in journals to advertise their services.

    There may be a concern that this would leave journals at risk of hiring under-qualified reviewers, but the suggestion of linking to their publications would surely mitigate this. If they lacked publications, or if an editor felt unable to judge their competence to review based on reading these publications, the contact details of a former PhD supervisor might also be included. It couldn't take long for an editor to fire off an email saying: "Is so-and-so qualified to review a journal paper on subject X?". This would surely be consistent with keeping to tight deadlines.

    Reply
  23. John Fischer

    Despite some exceptions, such as (among others) Nous, PPR, and Philosophers' Imprint, we have a real problem in our profession. I have had my share of "interesting" experiences with journals over the years, but I have been absolutely shocked to read in various threads on this blog and others, including the Garden of Forking Paths and Certain Doubts, about 17-month waits from journals (and more)–with no comments, or worse, hostile or unhelpful comments. This is devastating to the whole idea of an intellectual community with genuine discussions of issues of shared interest. As has been noted, it hurts all of us, but especially younger philosophers.

    I've thought about this situation for some time. Despite my respect for David Velleman and my sympathy with some of his worries, I wish to express my strong support for the serious consideration of something like David Chalmers' proposal (suitably revised, if necessary). It seems to me to be on the right track, and it also seems to me to be possible to implement it. Perhaps this is the kind of reasonably specific, clear, and constructive proposal that the leadership of the APA could consider.

    Reply
  24. Thom Brooks

    While I genuinely love PhilPapers, I am not sure I would use a referee database. There are already other databases out there that have worked extremely well at finding referees, both amongst more established figures and newer members of the profession.

    In any event, I would oppose adding contact details of past teachers: if these were important, then perhaps we should only contact the past teachers.

    Reply
  25. Berit Brogaard

    As an associate editor of Erkenntnis I can say that I am in favor of both the initial proposal and the database for willing referees. The initial proposal is very much in line with Erkenntnis's current procedures. The standard refereeing time currently given to referees for Erkenntnis is 28 days. The time from submission to decision sometimes exceeds three months but this is primarily due to the fact that referees wait too long to decline to referee or accept the invitation to referee but never get around to it and then decline only once they get the reminder 30 – 40 days later. I do not know how to prevent this latter problem.

    I certainly would use a referee database, especially one which is linked to publications. A referee database would not increase the risk of hiring under-qualified referees. I would never choose a referee merely based on the fact that their name appears in a list of references or a referee database. In far the most cases in which I invite a person to referee a paper I am already familiar with the referee's work. In other cases I get advice from friends or colleagues. Having a referee database would not change any of my editorial practices but would only speed up the process.

    Reply
  26. John Ongley

    David Chalmers wrote: "I see that there is a journal editor meeting coming up at the Eastern APA" and Christopher Hitchcock wrote: "I think it would be great if … the association of journal editors got on board".

    Gentlemen: What editor meeting? Where do you see this? And what "association" of journal editors? Can you provide links to these?

    Reply
  27. Stewart Cohen

    I agree with most of what Dave says, though I share David V's worry about a one month deadline discouraging people from taking on referee assignments. Dave says that if the referee does not comply, the editor can simply move on to a new referee. But this is often not so simple. As a journal editor, I feel my greatest responsibility is to do everything I can to make sure that a referee is well-qualified. As an author, I know very well the frustration of receiving an incompetent referee's report. But finding well-qualified referees can be a difficult and time-consuming task.

    Mostly, we need a change in the way most people view refereeing. Everyone in the profession depends on the efficient operation of the journals. So we all have a responsibility to contribute to this. But many people do not take this responsibility seriously. Being busy is not a reason to turn down a request to referee a paper. We're all busy most of the time. I understand that sometimes people are unusually busy, or already burdened with referee assignments, but I doubt that this accounts for the rate at which people decline to referee. Some people don't even bother to respond to requests to referee a paper, even after repeated reminders. Even when one has a good reason to turn down a referee request, one can help out by suggesting some alternative referees. In the letter I send to potential referees, I ask that in the event he or she declines, to please suggest alternative referees. I put this in bold letters. Yet less than 50% comply with this request.

    I worry it will be difficult to change the way people view their refereeing responsibilities. But I think agreeing on a set of standards and publicizing them will help. So I welcome efforts to do this.

    Reply
  28. Thom Brooks

    John:

    The association is "the Association of Philosophy Journal Editors" (APJE) co-chaired by Carol Gould and I. As noted before, we're holding the APJE relaunch at the APA-Eastern, specifically, 28th December from 5.15-7.15pm. The Eastern is in NY this year. We do not yet have a website.

    Reply

Designed with WordPress