Leiter Reports: A Philosophy Blog

News and views about philosophy, the academic profession, academic freedom, intellectual culture, and other topics. The world’s most popular philosophy blog, since 2003.

  1. Fool's avatar
  2. Santa Monica's avatar
  3. Charles Bakker's avatar
  4. Matty Silverstein's avatar
  5. Jason's avatar
  6. Nathan Meyvis's avatar
  7. Stefan Sciaraffa's avatar

    The McMaster Department of Philosophy has now put together the following notice commemorating Barry: Barry Allen: A Philosophical Life Barry…

A site for matching journals and referees?

Brian Leiter

Nicolas Espinoza, a philosopher at Stockholm University (who you can reach at nicolas-dot-espinoza-at-gmail-dot-com) writes:

Like many other junior philosophers (and i suspect senior ones too) I am frustrated by the long turn around time for submitted manuscripts to journals. So now I plan to do something about it. 

My impression is that one of the main problems is that editors have a hard time getting hold of referees who are willing to write prompt reviews. To me this is surprising since I know of many people who never get any refereeing jobs but who would be more than willing to help out. Also there is no real incentive for referees to put in the work.

My idea is to build a site which lets editors get in contact with referees, a match-making site of sorts or a referee brokerage site if you will. The site will enable philosophers to sign up and upload their academic profile (what kind of papers they are willing to referee). Editors will be able to post assignments and "auction" them off to the referees who promise the shortest response time. There will be a scoring system which specifies what journals you have gotten assignments from and reports on how you did in terms of quality etc. There will be rankings of best referees etc… The details are not worked out fully yet. One idea I'm not fully clear over yet is weather to charge the publishers for broadcasting assignments on the site and letting some of that money reach the referees (the publisher will pay more for shorter return times for example). I am very open to suggestions and would be very glad to hear what people have to say about this. Is there any specific funtionality that would be nice to have? 

Thoughts from readers?  You can also e-mail Dr. Espinoza directly, but I have opened comments here as well.

Leave a Reply to Ole Martin Moen Cancel reply

Your email address will not be published. Required fields are marked *

26 responses to “A site for matching journals and referees?”

  1. Steven French

    Its an interesting idea but many journals, like ours (BJPS), already have a pretty good (and fully searchable in terms of areas of interest) database of referees. And although we do sometimes have trouble finding referees for certain papers (in one recent case going through no less than 8 names before finding someone), the real drag is simply the time taken – to either let us know that the person concerned can/cannot referee the paper, but more often, and not surprisingly, to submit the report. In my view that is *the* major determinant of turn-around time and short of putting to one side the fact that 'referees r us' and sending round the guys with the electric cattle prods, I'm not sure there's much that can be done about that. We're all ridiculously busy and sometimes even with the best will in the world, its impossible to find the however many hours necessary to carefully read a paper and write a useful review. Policies such as ours and Thought's of offering quick responses but with short/no reports help to a certain degree and perhaps this mega-database will give us a wider choice but the basic problem will remain, I suspect.

    Reply
  2. grad student

    I can think of another simple solution to the problem: Allow submissions to multiple journals at once, with the option to withdraw the submission should one of the journals accept the paper first. (This practice is already common in Law, at least, showing it's not infeasible.) It's an absurd combination of circumstances for professional and aspiring philosophers that: acceptance rates are low, turnaround times are often pathetic, and we are only allowed to submit to one journal at a time. As a grad student under increasing pressure to publish, it is aggravating that I may wait 8, 12, or 15 months to receive a desk rejection, just so I can submit some paper to a second journal. Are journals *trying* to make things unfairly burdensome?

    Reply
  3. David Sobel

    Without there being an auction such that the assignments go to the referees who are willing to do the report in the shortest period of time, I don't think that it will do much to reduce review times. But it seems to me that journals should not just give the refereeing job to the person willing to review it in the shortest amount of time. I suspect that journals are far more concerned with whether a potential referee is someone whose judgment they trust than they are with whether a potential referee can promise a report in 4 or 8 weeks time. I suppose that the ranking of referees in terms of their quality is supposed to help alleviate this worry. But I'm not sure it will. One editor may not trust the judgment of other editors with regard to the quality of the referee reports (and judgments) they receive. Also, I'm a bit turned off by the idea of being judged by the quality of my reports. I put a lot of effort into them and thereby render my best judgment. If they don't trust my judgment, then they shouldn't have asked for it in the first place. Also, I get more and more editors asking for less detailed reports but quicker response times given that they trust my judgment and don't need me to give a detailed defense of my judgment but only a very brief explanation of why I came to that judgment. So if referees were all required to defend their judgments in the way that would allow editors to rank the quality of those judgments, this might actually lengthen the review process.

    Reply
  4. Steven French

    No we're not, honest! But look at your proposal from our point of view: we receive a submission, put a lot of time into reading it (yes editors do that) finding referees, nudging them to send in their reports, coming up with a recommendation/editorial decision based on what are sometimes divergent or even contradictory reports, sometimes having to go and find a third or fourth referee to help adjudicate … and then discover that its also been sent to another journal whose referees aren't so tied up with admin (or whatever) so it ends up being accepted there. Cheers, thanks for that! Seriously *long turnaround times are not (typically) the editors' fault*. But with increasing admin loads (at least here in the UK) the situation is unlikely to get better; hence the attractiveness of a quick decision/short report policy.

    Reply
  5. Eli Weber

    I'll just chime in that I'm not sure how much the "quick decision/short report" policy is helping. I just got such a rejection from a very prominent journal, which took nine months to receive. Is that really an improvement? It seems to me that something more than this policy is in order. As grad student notes above, those of us who are preparing for the market are under incredible pressure to publish, and this kind of turn-around time is anathema to that expectation.

    Reply
  6. Matt

    Allow submissions to multiple journals at once, with the option to withdraw the submission should one of the journals accept the paper first. (This practice is already common in Law, at least, showing it's not infeasible.)

    I'd be slower to draw this conclusion. Law reviews allow multiple submissions, but they also are not blind or peer reviewed. In fact, CVs are submitted with papers, which are picked by an army of students, often using name recognition, letter-head, etc. to make decisions. I assume that philosophers want to avoid this situation. Given this, it's at least not obvious that allowing multiple submissions in philosophy is a lesson we should learn from law. Some much weaker form of multiple submissions might be possible or desirable, but we'd not learn that from looking at law reviews.

    Reply
  7. Fritz Allhoff

    At *Public Affairs Quarterly*, we just don't have a hard time finding referees. Partially, this is due to our outstanding Managing Editor (Nathan Stout), but it's not something that takes long to figure out by looking at the footnotes and/or Google. I think the plug for this site should be that there are better referees available, especially young faculty who might not be showing up in footnotes or Google yet. But then the pitch should be that way; finding reviewers (at all) just isn't a problem. (Or at least not for our journal; I'd be interested to hear if others have problems in this regard. I suspect that difficulties in finding a reviewer probably has more to do with unpublishable papers than with journal functionality…)

    Reply
  8. Jozef Muller

    My only comment is that people from other fields that I speak to find the turn-around times common in philosophy unthinkable. Not that they are a bit longer (by say 30-50%), but that they are 3 to 5 and more times longer. But even in philosophy, there are some journals that have a much better reputation than others: so it cannot be just in the nature of philosophy that it takes longer to assess articles than, say, in geography.

    Reply
  9. Anon Grad Student

    Even if multiple submissions do not work, Nicolas' idea of a website aims to improve better communication between authors and referees. Referees can update the authors on how far they are into the review process. Graduate students may appreciate this reference–if you order pizza online from PizzaHut.com or Dominos Pizza, a webpage appears on how long the delivery will take and where the pizza is in the process. A website that editors and reviewers inform authors about submissions could be helpful. The additional information, similar to online delivery pizza, would say when the paper has been submitted to each reviewer. Now, this would be extra work for referees and editors, but this is merely one of many possible strategies to improve such communication.

    Reply
  10. SC

    One problem that this proposal might solve is the uncertainty about how to volunteer oneself as a referee. Or are there conventions out there that many people are just unaware of?

    Reply
  11. Nathan Stout

    As Fritz said, we haven't had too much trouble finding reviewers at PAQ. A site that potential referees could submit CV's to as well as a list of paper topics that they would be available to review would be nice, and it would certainly make finding reviewers a little easier. However, the times that I do have difficulty finding a reviewer typically happen because I have a paper on a topic that not many people have written on. Thus, I'm not sure how helpful it would be since there are not likely to be many people listed on said site that have these topics listed as one's that they could referee for, anyway.

    That said, I can identify two main causes for extended review times. The first, and most frustrating, is simply that potential referees often do not respond to emails. I'll send a review request that will go unanswered for close to two weeks at which point I have to find a different reviewer and hope that he or she is better at responding to emails. If this happens two or three times, then you've added an unnecessary 3+ weeks to the review time. The proposed site might help with this in that those people who have put their names in the hat are likely to respond when solicited. The second delay that we often have is the same that others have already mentioned – reviewers just take a long time to finish their reviews. This hasn't been too big of a problem for us so far, but I'm not sure the proposed solution (i.e. give the job to the fastest reviewer) is something that I'd be interested in as an editor. I'm more interested in getting the most talented philosopher to review the paper. I generally don't even ask anyone who hasn't published multiple papers in the same field as the submitted paper, and I often ask the most well-known philosophers first. As an editor, the goal is to get the best papers that you can into your journal so that other talented people will be more likely to submit their papers to you. This is how you increase your readership and your reputation. So, all that is to say that I wouldn't be likely to solicit reviews from people based on their proposed turnaround time.

    Reply
  12. Rebecca Kukla

    The weird thing about this proposal and discussion is it seems premised on the idea that people are vying to be referees and motivated to have it known how responsible and effective they are at doing it. I think I do a pretty timely and thoughtful job of refereeing and I try to say 'yes' to referee requests at least 10 or so times a year, but the LAST thing I want is MORE requests! My heart sinks whenever I get one. And most friends I know are the same. We do it out of duty and there's basically no reward for it. So why would I want a reputation online as a good go-to person? 'Willing' to write prompt reviews ~= clamoring to do so.

    Reply
  13. Mark van Roojen

    What Matt said and what Rebecca said. As Matt points out the possibility proof from law journals shows we can have multiple submissions and nonblind refereeing by grad students. And as Rebecca says, most of us who referee aren't eager to do even more.

    An information clearinghouse might be helpful, for example when you just can't think of anyone to do the job. But in the little bit of editing I do, it really matters to me that I trust the person's judgement. And I want that person to articulate their thoughts for me and the author. Refereeing is not just a matter of getting a decision from someone or other. So I myself would be unlikely to go to the proposed site to find referees. If I really can't think of anyone I might look through PhilPapers for authors who work on relevant topics. Or I'd ask advice from people who know more.

    I'm not defending the long time it often takes to referee articles. But at least one cause that I suspect is somewhat special to philosophy is that we have too many people needing to publish papers for job-market related reasons. This makes people submit papers that aren't ready, and to submit more papers than they otherwise would. We have probably seen a submissions increase by hundreds of percent in the past two decades, without a similar increase in eligible referees. My guess is that multiple authorship in the sciences helps take that kind of pressure off young people in those fields – a grad student will likely have her name on a few papers by the time the job market rolls around. I've heard that in some other humanities fields people pay to get published ('subvention' is the word, I believe). So my guess is that cross-field comparisons are somewhat misleading here.

    FWIW, I would probably stop refereeing if philosophy standardly went to multiple submissions. I do my best to write helpful comments on even papers that I think should be rejected. If I thought that this paper was in the hands of ten other people at other journals, I would not find that worth my time. I've already refereed more papers than I'll submit in five lifetimes, so my dues are paid.

    FWIW . . .

    Reply
  14. Neil

    Like Fritz and Nathan, I have no trouble finding referees. Philpapers allows one to search on keywords and turn up multiple candidates in seconds. I look for those who have published on the topic in the past 5 years. The problem, as Jozef points out, is the time referees take. The difference is not, as he says, 30-50% longer, at least wrt the disciplines with which I am familiar; it is 300-500% longer. For reasons I don't understand, the convention has appeared in philosophy that it is fine to ignore the papers one is refereeing for between 8 weeks and 8 months (depending on the prestige of the journal) before one does anything about it. I just don't believe that people are too busy for weeks on end to referee. Why in philosophy alone? While you were *too busy* did you do your own research? Then you weren't too busy. In any case, you won't be less busy in n weeks than you are now (google "planning fallacy"). Read the paper and write the review. If you don't have time, then don't accept the request. And if you don't have time to referee, you don't have time to do research, so don't submit anything either.

    Reply
  15. Neil

    In the sciences with which I'm familiar, the referee load per referee is *far* higher than in philosophy. A senior neurochemist I know has a policy of refereeing 50 papers per year. Granted, science papers are shorter, on average, and formulaic in structure. In some ways, that might make them easier to referee. At least the argumentative structure is clear (does this methodology really address that question? Is it sound? Does the data support the claim?) But the fact that referees churn through so many more papers than in philosophy is largely because of different expectations.

    Reply
  16. Neil

    Oops, sorry, I made a mistake. The scientist I had in mind has the policy of refereeing 100 papers, not 50 papers, per year.

    Reply
  17. Kenny Easwaran

    One thing that might be useful about this site is actually perhaps the opposite of what has been suggested. Many people feel they get too many refereeing requests to be able to do them all effectively, and sometimes limit the rate at which they are writing them. The advantage of a centralized refereeing database would be that journals can choose someone who wasn't just asked a few days ago by a different journal. I suspect part of the problem on both sides is that lots of journals end up asking the same people over and over again, while there are lots of other people that are also willing and able to write reports but the journals don't yet know. Having a centralized site might help the journals allocate the work more efficiently.

    Reply
  18. Greg Janzen

    "Seriously *long turnaround times are not (typically) the editors' fault*."

    Then why do some philosophy journals have notably fast turnaround times (e.g., AJP, European Journal of Philosophy) while others don't? One would think that the discrepancy is due to the journals' different editorial policies. After all, it strains credibility to think that the reason journals with comically slow turnaround times (e.g., Mind, JOP) continue to have comically slow turnaround times is that they persistently get stuck with referees who can't seem to muster a report in less than 9 months.

    Reply
  19. Richard Baron

    Previous commentators on this thread have put forward arguments for, and arguments against, multiple submission. Arguments on both sides are strong. The reason for this uncomfortable result seems to be that the point of divergence of the tracks along which a paper runs is the author. He or she sends the paper to one or more journals. The paper then works its way along a track within the publisher of each journal, and the tracks (if there are two or more) never meet again.

    How would it be if we located the point of divergence further down the line? That is, each paper would be submitted to a single pool of journals (say, a number of journals in ethics, or in the philosophy of science). It would then get reviewed in the pool, and articles, with their reviews attached, would then be available to all participating journals to fish out of the pool and publish.

    That would be a significant change to the structure of journal publishing, and it would have all sorts of other effects. The editors of individual journals would lose some autonomy, and journals might lose some of their distinctive styles. There would also be a change in people's perceptions of the prestige of particular journals, once they were seen as not entirely free-standing; but since it is the content of a paper that matters, not the brand name under which it is published, I would not attach any importance to that. In any case, we now go straight to the papers through our computers, and barely notice the titles of the journals.

    Reply
  20. Aidan McGlynn

    My impression from submitting to journals which allow you to track the progress of your paper is that referee assignment is reasonably often a point at which a paper gets snagged, sometimes for much longer than it takes the eventually-assigned referees to do their thing. Waits of 4-6 months don't seem to be at all unusual. And this does seem to be something that varies somewhat from journal to journal, with effects on their typical turnaround times. Such is my impression, at least, from my own experiences and from talking to others at around the same stage in their careers.

    I don't for a second mean to deny that slow referees are often the reason for long turnaround times, nor do I wish to point any fingers at editors. But I'm a bit dubious of the suggestion that slow referees are the only/the principal factor leading to slow turnarounds. I'm not sure what to make of the proposed website, but I wouldn't like to see the claim that there's a genuine issue here dismissed too quickly.

    Reply
  21. David Chalmers

    just saw this. at philpapers we're currently working on a database of philosophers (philpeople) which among other things will serve as a database of potential referees for editors to use. so this will have some of the functionality of the system suggested here. it won't have any sort of bidding system, though.

    Reply
  22. Ole Martin Moen

    Here’s a proposal: When a referee has assessed a paper, she should be able to accept it – not only for the one journal to which the paper was submitted – but also for other journals for which she referees. A response, then, could sound like this: ”Sadly, I cannot accept this paper for publication in Journal A, but I do accept it for publication in Journal B.” This could potentially save us a lot of time and effort.

    Reply
  23. Nicolas Espinoza

    This is very interesting! Does or will philpapers or philpeople make available any kind of api?

    Reply
  24. Nicolas Espinoza

    Thank you all for your most valuable comments! This has given me a lot to think about. I am in the process of raising funds for said project and if all works out development will commence some time in august. Until then I welcome any further suggestions you may have. All the best Nicolas

    Reply
  25. A Mod

    This practice is not common in law. Several law review journals (at least in Canada) do not accept simultaneous submissions.

    Reply
  26. Diana Constantinescu

    I think it is a nice initiative. It is doable as well. Philpeople could supplant this need, but some auctioning + some sort of rating (for how well and/or fast referees work) could definitely improve the process.

    Reply

Designed with WordPress