Leiter Reports: A Philosophy Blog

News and views about philosophy, the academic profession, academic freedom, intellectual culture, and other topics. The world’s most popular philosophy blog, since 2003.

  1. Fool's avatar
  2. Santa Monica's avatar
  3. Charles Bakker's avatar
  4. Matty Silverstein's avatar
  5. Jason's avatar
  6. Nathan Meyvis's avatar
  7. Stefan Sciaraffa's avatar

    The McMaster Department of Philosophy has now put together the following notice commemorating Barry: Barry Allen: A Philosophical Life Barry…

Transparency about refereeing

Brian Leiter

MOVING TO FRONT AGAIN, ORIGINALLY POSTED NOVEMBER 1–QUITE INTERESTING SET OF COMMENTS; MORE WELCOME

A junior philosopher elsewhere writes:

I've been meaning to write to ask if you might be willing to discuss journal review practices again on your blog. Some of your prior posts on the subject have been so useful. It would be good practice for journals to make much more explicit whether they are (and historically have been) triple blind, double blind, or neither, including publications based on conference proceedings (I'm thinking of the "Oxford Studies " series, for example). The lack of transparency is troubling, especially given that journals seem to change their review practices over time. Several publications I assumed were triple blind turned out not to be; and as a young scholar, one may not want to waste one's time with journals that are less than fully blind. It might also be worth pointing out that a publication's reputation in the discipline should be based on complete and accurate information regarding its review practices. 

"Triple blind" means, I take it, that the editors who pick referees do not know who the author is; the referees do not know who the author is; and the author does not know who the referees are.   "Double-blind" (author and referees are unknown to each other) is fairly common I think; my impression is many of the Oxford Studies volumes are single-blind (authors don't know the referees).   It would be interesting to know which journals are now triple-blind, so feel free to post links in the comments (or to comment on any related aspect of this issue).  Submit your comment only once, it may take awhile to appear.

 

Leave a Reply to Brian Leiter Cancel reply

Your email address will not be published. Required fields are marked *

34 responses to “Transparency about refereeing”

  1. Sebastian Lutz

    Ergo has a triple-anonymous review process (https://www.ergophiljournal.org/review.html )

    Reply
  2. B

    I had one experience with a journal using triple blind. I know this because I was asked to review MY own paper (though I was the second author on it). I declined. So triple blind is used (though this was not, strictly speaking, a philosophy journal).

    Reply
  3. Enzo Rossi

    European Journal of Political Theory has been triple-blind since we took it over in 2014. The acceptance rate is currently 9% (up from 8% last year).

    Reply
  4. Peter Furlong

    Res Philosophica uses triple-blind review. Here is the statement from their page:

    All unsolicited submissions are triple anonymous refereed: The author is anonymous to both the editor and the referee, and the referee is anonymous to the author. (We take anonymous review seriously; please see the instructions on how to prepare manuscripts for anonymous review.) If an editor can determine the identity of the author, no matter who the author, she will recuse herself from the review process entirely, and a suitable replacement will be found.

    http://www.resphilosophica.org/policies/review/

    Reply
  5. MM

    Adrian Piper has a list (don't know how comprehensive it is, haven't gone through it) of the policies of each journal:

    http://www.adrianpiper.com/berlinjphil/philosophy-journal-paper-submission-policies-2018-table-1.shtml

    Reply
  6. Eugene

    Both Oxford Studies in Metaethics and Normative Ethics are double blind. Not sure about the other Oxford series, but I believe most of them are as well.

    BL COMMENT: If anyone can provide a link supporting this, that would be great.

    Reply
  7. joe

    It's been my understanding that Oxford Studies are basically edited volumes in which each editor more or less decides how he wants to do things – if and when and how to engage other reviewers besides themselves.

    Reply
  8. David Wallace

    I'm far from convinced that this is something junior people should be too concerned about. There is basically no good evidence that double-blinding reduces bias* and if the theory-based concern
    is that unblinded journals will favor big-name philosophers, in practice articles from big-name philosophers are usually recognizable as such even through blinding. Things like turnaround time and demonstrated willingness to publish outside a narrow style are likely to be more significant.

    *I found

    https://blogs.plos.org/absolutely-maybe/2015/05/13/weighing-up-anonymity-and-openness-in-publication-peer-review/

    to be a good source on this and a good route into the literature).

    Reply
  9. Caleb

    Oxford Studies in Medieval Philosophy is committed to processing submissions "through a triple-blind refereeing process, to the fullest extent possible." https://spot.colorado.edu/~pasnau/osmp/notes.pdf

    Reply
  10. Charles Pigden

    Thought is triple blind. I have been asked to referee paper by a student of mine who the subject editor, who knew us both, knew to be a student of mine, thus proving that the editor was unaware of the author’s identity. (Naturally I recused myself)

    Does Res Philosophica’s policy, according to which an editor automatically recuses herself if she can guess at the identity of an author, apply to referees? If so it has some obvious adverse consequences. In small subfields (and in large subfields dominated buy a couple of prominent authors with distinctive philosophical styles) it may well be impossible to find a competent referee who cannot guess at the identities of at least some authors. For example, were I to submit a paper on the philosophy of conspiracy theories to some such journal as Episteme, any competent referee would be able to guess it was me after reading the first couple of paragraphs. In the much larger subfields that he dominated I don’t suppose that any competent referee could have failed to recognize a paper submitted by the late Jerry Fodor. In these kinds of cases it is relatively easy for a competent referee to see through the three levels of blinding which means that referees who cannot do this are unlikely to be competent. Thus the policy could lead to a situation in which every competent referee felt honour bound to recuse herself when faced with a paper by a well known author. This would not a desirable outcome.

    Reply
  11. Joona Räsänen

    Journals that say are triple-blind: Philosophical Review, Mind, Nous, Journal of Philosophy, Philosophy and Phenomenological Research,Philosophical Quarterly,Philosopher's Imprint, Analysis, Canadian Journal of Philosophy, Pacific Philosophical Quarterly, European Journal of Philosophy,Journal of American Philosophical Association, Ethics, Philosophy and Public Affairs, Journal of Applied Philosophy, Journal of Social Philosophy.

    BL: Really? All of these? Links?

    Reply
  12. Joona Räsänen

    Yes. I have collected this (and other relevant info for authors) from the websites of these journals (at the guidelines/instructions to the authors sections). You can view and download my file at my academia.edu page.
    https://www.academia.edu/40348800/Where_to_publish_in_Philosophy_Ethics_and_Bioethics

    Reply
  13. Brian Leiter

    Thank you!

    Reply
  14. Nick Laskowski

    I reached out to the editor of Oxford Studies in Metaethics, Russ Shafer-Landau, directly. I doubt he'd mind me posting our exchange, so here it is. It would probably be good to amend the original post for those not keeping up in the comments. It would also probably be good to update the journal lists from Adrian Piper and Joona Räsänen.

    Nick L
    Nov 1, 2019, 3:33 PM (19 hours ago)
    to Russ

    Hi Russ,

    There's a discussion happening here at Leiter Reports about the reviewing practices of journals and the like. There seems to be some disagreement over whether the various Oxford Studies volumes are blinded (and to what degree), including OSME. That got me thinking, especially since this is the sort of information that I'll have to provide to an RTP committee here at CSULB eventually. I've been under the impression that OSME is double blind (abstract writers don't know referee identities, abstract referees don't know author identities). Is that right?

    Thanks! I hope all is well!

    All best,
    Nick

    KEYWORDS:
    Primary Blog

    Reply
  15. David Velleman

    Why is it considered a sign of sound editorial practice that an author is invited to referee his/her own paper, or that a referee is invited to referee a paper of his own student? In those two cases, the invitations were declined. Would such invitations always be declined? I doubt it. Not everyone who recognizes the work of a friend, colleague, or student can be counted on to decline a refereeing request. *Someone* at the journal — someone knowledgeable about the field — should know the identity of the author so as to avoid inviting referees who are known to be colleagues, friends, former co-authors, teachers, or students of the author. That said, the decisions about whether to referee a paper and whether to accept it should be made in the dark.

    Reply
  16. Charles Pigden

    To David Velleman
    Well being asked to referee a paper by one of my students (and someone known to be one of my students by the subject editor) is a proof of triple-blindness on the part of Thought and if triple-blindness is a sound editorial practice it is a proof of that. But actually I am pretty agnostic about the 'if'. Is triple-blindness the acme of editorial integrity? Damned if know. What I am sure of is that it does not guard against the problem of its often being relatively easy for the referee to guess the identity of the author or for the author to guess the identity of the referee.

    Reply
  17. David Shoemaker

    It's important to remember that the Oxford Studies series that are connected to workshops have two stages of review, first at the abstract vetting stage, and second at the paper review stage (post-workshop). This typically means a variety of refereeing practices along the way. For my Oxford Studies in Agency and Responsibility, the abstract-vetting stage has become triple-blind (whereas in the early years it was single-blind, then it became double-blind). But this yields a subset of viable papers for the workshop that is nonetheless still too large, so I usually have to cut down to the final program from there, based on a wide variety of considerations, including program balance (across many dimensions). At that point, successful abstract-writers are guaranteed only a presentation at the biennial workshop; the paper itself is only "provisionally forthcoming" in the volume, although it must also be committed to being vetted for that volume (that is, it cannot be submitted elsewhere in the meantime). After the workshop, the papers are revised in light of comments from workshop attendees and me (a fully-sighted process). Once submitted to OUP, there is a single-blind refereeing process, with the entire proposed volume reviewed by two referees who know the authors, but who are unknown to them (and to me — so is it 1-1/2 blind at that point?).

    Reply
  18. Matías Vernengo

    I don't know in philosophy, but in economics the issue is that even when a journal is double blind, papers have been published as working papers and it's relatively simple to know who the authors are. For editors that means that it's hard to keep that part completely blind even if one wanted to.

    Reply
  19. Charles Pigden

    It is true that it is often easy to find out an author’s identity by a bit of judicious googling. But in philosophy there is a sort of honour code which says that you’re not supposed to do this. The thing is that even if you righteously refrain from looking and keep your googling fingers at a standstill it is often easy to guess at the identity of an author with a sufficiently distinctive style. Indeed the problem is not so much that you *can* guess at some authors’ identities as that it is difficult to refrain from doing so. ‘Oh that’s X’ you say to yourself after a couple of couple sentences.

    But returning to the OP, is this something that young philosophers need to worry about? Perhaps, if they work in a small subfield and have managed to acquire the dislike of one of the subfield’s panjandrums. But otherwise it is not likely to be an issue. It may be that big names are likely to be recognized by referees and are consequently more likely to be published. This diminishes the ‘space’ available in the journals for junior philosophers to strut their stuff. If famous old philosopher X gets published in the journal of Z Studies as a result a referee-recognition then it is a little less likely that unknown philosopher Y will be published in the same journal. But it is only a *little* less likely and to my mind it really isn’t worth worrying about. Referees may have bias in favour of those they know and respect but I don’t think that many of them have a bias against those that they *don’t* know. If the OP will forgive me I think that he or she is overthinking it.

    Reply
  20. Brian Leiter

    Philosophers underutilize "working papers" series, like that provided by SSRN. I plan to write something about that shortly.

    Reply
  21. Dean C. Rowan

    I'm eager to read what you write about this. You are likely aware of a species of over-reliance on SSRN by some legal scholars who are protective of their download counts. I don't know how widely legal scholars rely on working papers to stay informed or to monitor for preemption by others of their own work in progress, but I do know of scholars who resist the development of scholarship repositories at their respective law schools, because they don't want to dilute the SSRN count. Of course, legal scholars generally publish in journals that operate according to the bizarre, if historically entrenched, model of student curation and editing; there are relatively few peer-reviewed law journals. Maybe they view SSRN as a closer connection to their peers than the one afforded by student-edited law journals?

    When you say philosophers underutilize SSRN and the like, do you mean they don't rely on it to stay current? That they don't use it to promote their own work or to invite critical comments in advance of submission? Is that function served almost entirely by workshops, conferences, and the like? Is the number of philosophers so much smaller than the numbers of legal scholars or economists that it's easier for philosophers to keep up with others' work in progress?

    No need to reply here. I await the something you plan to deliver soon!

    Reply
  22. Steven French

    The BJPS follows a triple masking process. Leaving aside David Wallace's (entirely reasonable) point, the obvious downside, as has been pointed out, is that we then have to rely on the referees recusing themselves if there's a conflict of interest. We do explicitly ask about this in the invitation. I'm not sure returning to double-masking will help. David Velleman insists " *Someone* at the journal — someone knowledgeable about the field — should know the identity of the author so as to avoid inviting referees who are known to be colleagues, friends, former co-authors, teachers, or students of the author. " Really? I like to think I have a fair overview of who's doing what in those areas of philosophy of science that I work in but across the whole field, including philosophy of cognitive science, foundations of decision theory, not to mention philosophy of biology etc etc … no chance! And even in philosophy of physics, realism, models … I really doubt that given any particular author, I would know who their colleagues, friends, former co-authors, teachers, or students are! (I guess we could ask for a list …)

    Call me naive (don't all rush at once!) but sometimes we do just have to trust our colleagues (and if they don't tell us there's a conflict of interest and we find out, we can 'flag them' so they're not asked to referee for us in future)

    Reply
  23. E

    Is the concern that triple blind review risks authors reviewing their own papers a serious one? How hard is to eliminate or substantially mitigate such risks, by, e.g., appointing someone without final decision-making authority (even an internal editor) to double check whether the reviewer selected by the managing editor bears no obvious connection to the author? And to echo an earlier post, there is an honor code within the discipline of bringing one’s knowledge as a reviewer of an author’s identity to the editor’s attention. The honor code can and should be policed more carefully.

    My own impression is that a very small percentage of papers in philosophy (much smaller than the percentage that get accepted) are truly outstanding. The rank below consists of a large number of articles close enough in quality to make meritocratic discrimination difficult (*not impossible*). It is here I suspect knowledge of the author’s identity, stage of career, institutional affiliation, personal je ne sais quoi impacts program and publishing decisions. And I don’t find the armchair estimates offered above (in good faith, I trust) of the likelihood of editors being subject to such influences all that comforting.

    In any event, all that is being requested is transparency. No one is forcing Oxford Studies or any other publication to adopt triple blind review. Let a thousand flowers bloom! What would be helpful is a clear official statement from publications of their review practices, not a statement via email, or personal correspondence, or a comment on a blog post (given that what is true at a time needn’t always be true, among other reasons). A clearer formal statement, especially from journals which currently make ambiguous references to blind review, would greatly assist those of who care about this sort of thing, whether in deciding where to submit or what to read. How could more information possibly be a bad thing in this context?

    Reply
  24. Steven Nadler

    The Journal of the History of Philosophy was double-blind during my editorship (2010-2015), and remains so. (And I believe it has always been double-blind). I prefer this because (a) it prevents just the kind of mistakes that some of the comments above describe — sending a paper for review to the author, to his/her former student or graduate advisor, or to his/her current colleague; and (b) it offers a direct form of communication between editor and reviewer, something that is not only more personal and pleasant, but that helps alleviate the most difficult part of an editor's job, namely, finding willing and responsible reviewers. I think a potential reviewer is more likely to agree to review if asked personally by the editor than by either an automated review system (e.g., ScholarOne) or a managing editor.

    Reply
  25. Charles Pigden

    Steven French writes 'If [referees] don't tell us [at the BJPS] that there's a conflict of interest and we find out, we can 'flag them' so they're not asked to referee for us in future'.

    Isn't this like rewarding them for being corrupt?

    Reply
  26. Andrew Moon

    Here is info on CJP, which is wonderfully transparent about their process.

    https://www.cambridge.org/core/journals/canadian-journal-of-philosophy/information/peer-review-process?fbclid=IwAR0HZSG6P0Yn_OM684DP3EexQmWtgjUhjkwT8YrSeTiR8E7dGyzPaW3hPSM

    Reply
  27. Anon Tenured Faculty

    Thanks to those who've contributed to this helpful thread. I wanted to suggest that a lack of transparency shouldn't just affect a journal's reputation. The reputation of the editor, and even some authors, is on the line, too. A lack of transparency can allow editors and authors to get away with problematic arrangements, or let others wrongly accuse them of such arrangements when everything is actually on the up and up.

    What could the profession do about these issues? Well, academic philosophy could benefit from far more transparency as it bears on potential editorial conflicts of interest. Many editors have PhD students and recent graduates, for instance, and those junior philosophers are seeking venues to publish. But if editors do not recuse themselves when handling submissions from current or previous students, that information is highly relevant to the field. Search committees, for instance, should know whether or not an applicant who has published in that journal edited by his or her dissertation adviser was handled by said adviser/editor.

    Does triple blind review prevent academic publishing's version of cronyism and nepotism? No. Even if the adviser/editor doesn't see the author's name when making a judgment on the submission, the adviser/editor has already looked at earlier versions of the material.

    But there's a simple way for all of us to know what is going on with journals. Philosophy journals can follow commonplace editorial practices in the sciences, where the names of editors and handling editors for an article are routinely stated in the article itself. That practice makes editors inclined to recuse themselves when their involvement in a submission would be, or would appear to be, biased. Editors often have good reason not to handle submissions by certain authors: current/former students, friends, colleagues, siblings, lovers. Under the envisioned practice, it's not true an advisor could never serve as editor for students' work. But it does mean that we will know about it when it does happen. And then we can decide whether it's a problem or not.

    Reply
  28. Graeme Forbes

    When I was philosophy editor at Linguistics and Philosophy ('13 – '18) the journal had a double-blind system, and I think still does. I am not sure what practical difference it made, except that it was easier for junior philosophers to write to me to ask that the referees be told to get a move on (upcoming tenure review)! A couple of years ago I went to a session at the APA Pacific Division on the underrepresentation of women in philosophy journals. If I'm remembering correctly, statistical evidence was presented to the effect that in journals with a double-blind system, women were a bit less underrepresented than they were in journals with a triple-blind system. (Some data at http://women-in-philosophy.org, though for the refereeing point you will have to root around; perhaps "New data on the representation of women in philosophy journals: 2004–2015", Wilhelm, Conklin and Hassoun, PhilStudies 2018)

    I agree with those who have said here that the OP should not get too hung up on refereeing policies, especially the double vs triple blind distinction. It might make a difference on the very occasional all-else-equal situation, but even that I doubt. Anyone who isn't getting their papers published should be focussed first and foremost on the content of those papers: "if only they'd used a triple blind process they'd have accepted my submission" is *very* unlikely to be true, at least at L&P and others like it.

    Reply
  29. Danny

    Gabrielle Contessa has put together a collaborative google spreadsheet which collates information about philosophy journals and their practices (Including their peer review anonymity practices): https://docs.google.com/spreadsheets/d/e/2PACX-1vQpsiCzRS7W_caem84bJ8WQWVu1kbOnnKWfc7Z0zdUIqdCKzqw9M_dXxdTQ7Rao6ocbSm3RlCXYdn_1/pubhtml

    Reply
  30. Ned Block

    There is a form of transparency that is different from those discussed here: sending all the referees the reports by the other referees. I know many people who publish both in psychology/neuroscience journals and in philosophy journals who have noticed that the standard of refereeing in the science journals is much higher than in philosophy journals. Part of the explanation may be that many science journals have the policy of sharing the reports with the other referees. It really stands out when one reviewer hasn't bothered to understand what the submission is saying. Of course the other reviewers may not guess the identity of the wrongdoer but many reviewers say they find the prospect motivating.

    Reply
  31. Brian Leiter

    That's a really good idea, I wish it would become standard practice in philosophy!

    Reply
  32. E

    The complaint from OP needn't have much to do with whether their own papers are getting published. One can be concerned about cronyism and nepotism in the discipline on principle.

    Those dismissing the concerns raised by some of us have yet to seriously engage with the basic point that more transparency in the discipline would be a good thing. If 'double vs triple vs single blind' doesn't make much of a difference, as past/current editors of partially blind journals are inviting us to believe on the basis of their testimony alone, what's the harm in being clear about your preferred review process on your journal's website?

    Anon Tenured Faculty's suggestion that the names of editors and handling editors be stated in the article, as is convention in other disciplines, warrants serious discussion. I cannot come up with disadvantages to the proposal and so find myself wondering why we haven't already adopted the practice.

    Reply
  33. Charles Pigden

    It's not standard, but in my experience it is fairly common.

    Reply
  34. Charles Pigden

    Replying to E:
    'The complaint from OP needn't have much to do with whether their own papers are getting published.' True, but in fact it was. S/he wanted to know which journals were triple as opposed to double-blind because 'as a young scholar, one may not want to waste one's time with journals that are less than fully blind'. I agree with Graeme Forbes that whether or not a journal is triple as to as opposed to double-blind is unlikely to make much difference to one's chances of publication.

    —–
    KEYWORDS:
    Primary Blog

    Reply

Designed with WordPress