Leiter Reports: A Philosophy Blog

News and views about philosophy, the academic profession, academic freedom, intellectual culture, and other topics. The world’s most popular philosophy blog, since 2003.

  1. Daniel Kaufman's avatar
  2. Daniel Kaufman's avatar
  3. Daniel Kaufman's avatar
  4. Samir Okasha's avatar

    Gabriele Taylor was my undergraduate philosophy tutor at St Anne’s College, Oxford from 1989-1992. I remember her as an incisive…

  5. Mark's avatar
  6. Santa Monica's avatar
  7. David Wallace's avatar

How blind is “blind review” a grad student wonders?

Brian Leiter

A graduate student writes:

As a grad student trying to publish before I get my PhD, I've noticed something strange that happens every time I submit a paper to a journal.  Every time I submit, I get a hit on academia.edu that day. My academia.edu profile is pretty sparse and seldom gets hits, so this could be a coincidence, but that seems unlikely.  I also tend to get hits shortly after my paper changes from "Submission Approved" to "Editor Assigned."  It's possible that this is mere paranoia on my part, but it also seems possible that some of the journals I'm submitting to (about six or so) aren't as blind as they claim to be.  

This isn't something that an established philosopher would necessarily notice, as I'm sure you guys get hits all the time.  But I'm wondering if other grad students and junior philosophers have noticed this phenomenon.  It would be interesting to see if my guess has any merit. 

Readers?  Have others have had similar experiences?

Leave a Reply

Your email address will not be published. Required fields are marked *

31 responses to “How blind is “blind review” a grad student wonders?”

  1. Another grad

    I have had the same thing happen to me on several occasions, often a day or so before I hear a verdict on the paper (in my case, they've all been R&Rs or rejections so far). In one case, someone from the same city as the associate editor who was responsible for my (anonymous) desk rejection googled me the day before they emailed me. Most likely, people are preserving anonymity during the decision-making process and then googling your name afterwards out of curiosity, but it is disconcerting.

    Reply
  2. Sara L. Uckelman

    Unless the submissions are supposed to be triple-blind (that is, even the handling editor doesn't know the author's identity), I wouldn't be surprised at all to find the editor googling the submitter to find more information (in particular to lessen the chances of picking a referee who has to decline because of conflict of interest).

    Reply
  3. anonymous

    Theres is, of course, a difference between blind and double-blind. In the former, the editor but not the referee would know who you are, which would explain the hits on your site.

    Reply
  4. Paul

    The least cynical interpretation of this, to which I am at least initially inclined, is that it's a combination of coincidence with editors themselves looking up this person on Academia. That seems far more likely than six different editors all sending identifying information to the reviewers. If, however, this person has drafts of the papers, using the same title, on Academia already, then there's an even more simple explanation (reviewers are searching for the paper on their own) that doesn't question the blind review process.

    Reply
  5. Neil Easterbrook

    While my background is in philosophy & my discipline is literature, I can say that without any question no journal is as blind as it claims to be.

    I'm on the editorial advisory board of four journals, so I referee lots of papers each year (and, like many people, I also referee for other journals and university presses). Usually the MS is sent to me "blind," but often the journal's editor hasn't stripped author info from the properties data (as in doc or docx files); quite often the paper contains self-reference to other papers; and equally as often the paper's title also appears in an online cv, conference program, or academia.edu (etc) site. Often then, a few key strokes are all that's needed to identify an author, and I'm a curious fella.

    But generally it's not the journal editors who are at fault: it's the referees, like me.

    In my case, I don't quite see the importance of blind review, and I've signed every referee report I've ever written, even when I've had some pretty dismissive things to say in recommending rejection.

    But I also can't, in all honesty, think of a single instance where knowing the author's name (and rank, and school, and status) influenced my recommendation; I've recommend rejection of papers by full professors at exceptional institutions, and also recommended publication of high school teachers without advanced degrees (tho of course these are both test cases). I hear there's lots of logrolling in academic publication, but I've never once witnessed or participated in it.

    So even if journals aren't as blind as they claim, I'm not sure that (a) this is necessarily a bad thing, or (b) there's much we can do abt it.

    Reply
  6. Daniel A. Kaufman

    We've had this conversation before and it created a lot of hostility. All that I can say is that I spent quite some time as the Managing Editor of a journal back in the 90s, and blind referreeing was certainly not applied uniformly. Conversations with other managing editors suggested that this was not unusual in the business.

    If you think that a paper submitted by a Quine or a Davidson or a Rorty gets the same treatment as yours, I have a few bridges to sell you. (And yes, I did deliberately use two people who are no longer with us, so as not to get the inevitable — "Oh yes! All my papers were blind refereed.")

    Some of us may have been born at night, but it wasn't last night.

    Reply
  7. Anonymous junior philosopher

    I've had similar experiences. I get hits on the research page of website just prior to a change of date to "reviewer assigned" (a referee apparently declined to review the paper) and when my paper moves from "reviewer assigned" to "under review." I think it would be helpful if journals with double-blind review policies were to include a statement of these policies in their invitations to referees, along with a statement to the effect that referees are expected to strictly adhere to double-blind review.

    Reply
  8. John Rapko, PhD, Lecturer in Ethics and Critical Studies, California College of the Arts

    I'm not a graduate student, but I've noticed a similar pattern. As Hölderlin suggested, perhaps blind reviewers have one eye too many.

    Reply
  9. Another junior philosopher

    I once made the mistake of having a draft of a paper that I submitted to a journal posted on my academia.edu page. A few days after I submitted it, somebody searched google for the exact title of the paper (academia tells you the search terms people use) and found it on my page. The location of the hit (a relatively small city) also happened to be the location of a university, and that university happened to employ a relatively senior figure who works on the topic my paper was about. The paper was eventually rejected. (Although, in hindsight, it probably deserved to be, so I don't think any real harm was done in this case.)

    Reply
  10. Anon Grad Student

    Neil Easterbrook fails to take into account the highly significant possibility of implicit bias. Seeing that a paper is, say, written by a PhD candidate from a below top 30 university, rather than by a prominent professor, might influence a negative reaction even if the paper itself is good.

    In fact there really should be some systematic research testing just this possibility.

    Reply
  11. Steven Hales

    I'm asked to referee fairly often (always blind) and I have never googled a paper title in an effort to figure out the author. I don't want to know, except in the rare cases when it is a paper I'm recommending for publication and is on a topic I'm actively researching. Why can't even curious referees restrain their impulses?

    Reply
  12. Another grad student

    For grad students, is there any reason to be posting drafts or submitted papers online, either on a site like academia.edu or on one's own website? I think a topic like this one came up a couple of years ago on this site, but after reading a few of these comments, it seems like the possibility of negative bias (and other hurtful outcomes) outweighs any benefit of posting a paper online. It's not like posting it online for all to see is the only way to share your work in progress.

    Reply
  13. Dale Miller

    A hit on the very same day that the paper is submitted almost certainly comes from someone at the journal itself, presumably someone whom is allowed to know the author's identity, rather than a reviewer. Later hits may well be from reviewers, but that doesn't mean that the reviewers Googled before submitting their reports.

    Reply
  14. curious onlooker

    I'm surprised no one has pointed out the obvious: if you want your paper to receive a blind review, why on earth would you post a draft version on the internet? You can't guarantee that your paper will receive a blind review, but you can certainly prevent it from doing so!

    Reply
  15. Anonymous

    From 1 above: "Most likely, people are preserving anonymity during the decision-making process and then googling your name afterwards out of curiosity……"

    It is important that reviewers not do this. A paper that has been reviewed will sometimes be sent back to the same reviewer later for a "second round" of review, or if the first version received a "revise and resubmit". So if you look up the author's identity after the first review, you've subverted this process. Reviewers just need to refrain from being curious and leave things alone.

    Reply
  16. Syd Johnson

    I have never searched for a paper's title or author when I'm reviewing it. Doing so is just asking for trouble, and implicit bias is the trouble — you don't KNOW you have an implicit bias, and so you can do nothing to counteract it. That said, my research niche is fairly small, and I often have a pretty good idea about the identity of the author(s) based on the subject matter alone (and this usually gets confirmed if the paper is published). I hope that not knowing for sure goes some way towards removing the potential for acting on bias.

    Reply
  17. Franz Huber

    If you want your work to be reviewed anonymously, you must, of course, not list its title online, let alone upload a draft to your homepage or some repository!!!

    The same is true for listing talks on your unpublished work online, such as talks you give at conferences or other venues that make this information available publicly. If you really have to list these things, at least change the titles or replace them by general descriptions of the work in question (say, "paper on moral realism").

    Alternatively, remove the title of your paper when you submit it to a journal!

    Editors can only ask their reviewers to not search for the work they review online. Even if all reviewers followed this request, they would still come across other people's homepages and the relevant titles if they are experts in the relevant field.

    Making your work available publicly before it has been accepted for publication is your decision. If you want your work to be reviewed anonymously, don't make it available publicly. If you decide to make your work available publicly, then don't blame editors or reviewers if your work does not end up being reviewed anonymously: in this case it's your own fault.

    Reply
  18. Another junior philosopher

    Just to be clear, I don't disagree with the comments here to the effect if I really wanted my paper to be blind-reviewed, I shouldn't have posted it on-line ahead of time. Obviously this is true (although the thought didn't occur to me when I first put the paper on-line.) Frankly, I don't care all that much whether my papers are blind-reviewed or not. I would prefer it, but it's not that big of a deal to me. And, as I said in the original post, in this case I don't think it had any impact on the outcome. The point of my original post was not that I was the victim of some injustice. I wasn't. Brian just asked if "other grad students and junior philosophers have noticed [the] phenomenon" of getting academia.edu hits after submitting a paper, and I was reporting that I had.

    Reply
  19. Yet Another Anon Grad Student

    It may well be that most of these hits are from editorial staff, but there is still reason to worry about this considering many journals have a huge number of desk rejections (some at around 60%). The blind review process is going to be something of a moot point if there is a non-blind screening process prior to review. A lot of the top journals are supposed to be triple blind. It can be just as bad for an editor to look up a paper's author if he or she does this prior to giving a desk rejection. Though it is not problematic if it is just a managing editor who does this (i.e. someone who does not have the power to desk-reject).

    Reply
  20. Jamie Dreier

    Sorry to be thick about this, but is the evidence from the OP and others pointing to referees or editors googling papers to find the authors, or is it partly at least referees or editors who somehow already know the names of the authors googling them to find out more about them? (Both are disturbing.)

    I hope Dan A. Kaufman was speaking just of his own journal when he told everyone who expects their paper to get the same treatment that a famous person’s paper gets that they’re just suckers. At some journals all papers do indeed get the same treatment irrespective of fame or status of author.

    Reply
  21. Daniel A. Kaufman

    Maybe things have changed since the 90s, but I doubt it. And as I indicated, from my conversations with other managing editors, it was not uncommon.

    Reply
  22. anon ex-adjunct

    "But I also can't, in all honesty, think of a single instance where knowing the author's name (and rank, and school, and status) influenced my recommendation."

    It bears repeating that anecdotal evidence is generally worthless.

    Reply
  23. anon ex-adjunct

    "I'm surprised no one has pointed out the obvious: if you want your paper to receive a blind review, why on earth would you post a draft version on the internet? You can't guarantee that your paper will receive a blind review, but you can certainly prevent it from doing so!"

    Or, why on earth did you leave your car unlocked in such a bad neighborhood? You can't blame anyone but yourself that it was stolen.

    Key in here: editors and referees shouldn't be attempting to identify the author of a paper that's under blind review. Sometimes it will happen by accident, but then the editor or referee should pass the paper on to someone else.

    Reply
  24. A wise philosopher

    To those of you browbeating anyone who post titles of work-in-progress on their website: you seem to be forgetting that the same is done by anyone who (i) presents their work-in-progress and (ii) keeps an accurate, up-to-date C.V. on their website. Are we all now supposed to stop doing this, too, if we happen to desire that our work-in-progress be anonymously reviewed?

    Reply
  25. David Wallace

    I have to say that (as always when this topic comes up) I find it a bit perplexing how people think about publishing work on the internet. Look: if what you want is feedback on your work, circulate it privately among those who you think will be willing to provide it. If you make it openly and publicly available on archives (philpapers.org, academia.edu, arxiv.org) expressly designed to make publications widely visible, you are *publishing* it, and you should be doing so because you want your paper to be widely read, to be discussed, to be criticised, to be cited, and for all that to happen quickly, rather than to wait possibly several years for the paper to snake its way through the refereeing, copy-editing and publishing processes. (Accelerating those processes was the entire point of the original online archives in physics, and their preprint-circulation pre-internet ancestors.) So in that case, you should want the referees of your paper (who, ex hypothesi, are likely to be experts in exactly the field your paper is in) to be aware of your paper and to be able to critically engage with it.

    Of course, you might decide that, for you, the costs of forfeiting blind review outweigh the benefits of speedy and widespread circulation of your ideas and work. That's your call; if so, don't put the paper on an online archive.

    For what it's worth, I do refrain from Googling anonymised papers when I referee, but only on the contractual grounds that the journal asks me to and that I shouldn't accept referee requests if I'm not willing to abide by the terms and conditions of those requests. I find the whole thing pretty quixotic, though.

    (Don't get me started on "do not quote or cite" notices on publicly archived papers!)

    Reply
  26. Bill Wringe

    Comment 12:

    'For grad students, is there any reason to be posting drafts or submitted papers online, either on a site like academia.edu or on one's own website?'

    Academia.edu has recently introduced a feature that enables ypu to set up an on-line feedback session for your paper. I've had mixed results using these; but I can see how they might be useful for people – especially people who for one reason or another (family circumstances; lack of funds for graduate students; inconveniences of geography) are less abole to get feedback on their work at conferences.

    Obviously, well-connected people, in strong schools on the East Coast (for example) may not get much benefit from this kind of thing. But I'd suggest not instituting or reinforcing norms that make it harder for people who aren't in that situation not to level the playing field a little.

    I expect that some of those who are in this position will protest loud and long at this suggestion. I'd probably be given the dreadful label 'Social Justice Warrior' if I said what I thought of that.

    Dale Miller @ 13: We've already got one person on the thread saying that they go on-line to find out the author of things they're refereeing becuase they don't see the value of blind review(Neil Easterbrook @5), so we don't need to debate whether the phenomenon is occurring. There may still be a question as to how common it is.

    Neil Easterbrook @5: would your indifference to the blindness of the review process extend to cases where the journal you are reviewing has given an explicit commitment to blind review?

    If the answer's yes, I'd suggest that journals which give a commitment to blind review will take note of your comment, and don't send you things to review (that would obviously be no incentive to change your ways.)

    Franz Huber @ 17 (!!!): When I agree to review a paper, (blindly or otherwise), I'm undertaking a commitment to an editor and not – or at least, not in the first instance – to the author.

    If I give a commitment to an editor not to endeavour to find out the author of a paper, then I can't see how something the author does – even putting the paper online – releases me from that commitment.

    Some journals – such as the Europena Journal of Political Theory – explicitly ask authors to make that commitment when they agree to review. Perhaps more should; for myself,I've always taken that commitment to be implicit in an agreement to review; (and I would do so even if I were like Neil Easterbrook, and didn't think the practice had any value.)

    Clarificatory footnote 1: I've said that a referee's commitiment is, in the first instance to the editor, not the author of the paper. They may have some derivative commitments to the author. Does anyone have a direct commitment to the author? Yes: the editor – if they advertise that they practice blind review, they're committed to taking reasonable steps to ensure that's the case. One such reasonable step might be to ask referees not to Google people. A second would be to avoid asking, as referees, people who openly flout the norms of the practice.

    Clarificatory footnote 2: It may be that for all I say, you're a fool if you expect people working in the same profession – and professional competitors to boot – as you to stick to their commitments. But the fact that you're a fool does not in any repect mitigate others' obligation not to be knaves.

    Reply
  27. Another grad

    I took the conversation to be about editors and referees googling the names of authors to find out more about them–in any case that's definitely what I was referring to in my initial comment. I agree that the two kinds of behaviour are troubling in different ways.

    Reply
  28. anonymous phd student

    A few considerations:

    -I'd like to mention that there are some journals (e.g., the review of phil and psychology) that allow the author to suggest potential referees. Some personal experience about this: Academia once detected and revealed unusual traffic coming from the cities where the suggested reviewers had their affiliation. That happened around the day in which I received the outcome of a submission. One of the reviewers was rather happy with the paper as it was. The other one asked for quite major revisions. The unusual traffic is consistent with different hypothesis, but the timing provides some constraints.

    -I don't have any drafts under review or in progress that have been made available on the internet. I do this because I think that the risk of having ideas 'stolen' and being identified as a graduate student by journal referees is more serious than the benefit of showing to, say, hiring committees, what's in the pipelines (and I also think I can still show what's in progress in cover letters). Having said that, maybe I will sound overly idealistic, but I can still see reasons for being willing to share information/knowledge. Being passionate about a subject/topic often implies being willing to share ideas, especially when you think they're good ideas.

    -In some cases different papers (published, in progress, under review) are also 'contaminated', especially if these research outputs are part of the same project'. in particular, if you're working on very specific topics you might happen to use the same tags or labels to refer to the same people, groups, ideas. This suggests than in some cases not having drafts of papers under review available on the web does not guarantee that the extremely curious reviewer won't be able to make a safe bet about the author's identity. If the stalker is a talented stalker, you must be more careful than what some of the comments above seem to suggest.

    -But the fact that many people detect these patters through Academia suggest that the reviewers aren't probably too skilful stalkers or, more seriously, just don't care about the potential consequences of their actions. One possible consequence is this, I think. In light of the information offered by Academia about traffic, hits, etc., authors' trust in peer-review can decrease. This suggests that, perhaps, we should not just tell authors how silly was their decision to make research available before acceptance, but we should also tell reviewers that their behaviour is certainly not helping to promote trust.

    Reply
  29. Neil Easterbrook

    Re: implicit bias. Yes of course there's implicit bias in referring papers, and that's the case whether the review is "blind" or not. We all work to keep the principles of charity and distance doing what they can to combat own bias and sloppiness.

    But as long as the review is peer review, and rigorous, I don't see a scandal.

    You want a scandal, now here's a scandal: http://www.slate.com/articles/health_and_science/science/2015/04/fake_peer_review_scientific_journals_publish_fraudulent_plagiarized_or_nonsense.single.html

    Reply
  30. Neil Easterbrook

    Bill–

    Invoked by yr post, I've written a fairly long email to the main editors where I'm on the advisory boards of scholarly journals, asking both for their response abt the issues raised here and including my offer to resign if they think my practice unreasonable.

    But I do want to emphasize that I've signed every referee report I've written in the last 25 years; at least on my part, I've never invoked anonymity & never consequently pretended that I am without bias.

    And let's be clear abt the main point: blind review promises *only* anonymity–it does not promise fairness or non-biased judgment or anything else. If someone believes that the former necessarily entails the latter, that's the point where professors need to sit down with grad students, both to help them with their professional work and to disabuse them of notions that, perhaps, they ought to have dismissed quite early in their intellectual work.

    If disinterestedness remains a desideratum in our work, then alas it is a horizon never reached.

    Reply
  31. Yet Another Anon Grad Student

    We will never attain perfect justice or eliminate crime. Should we therefore disband the police force and the entire judicial system? Obviously not. Similarly, we may never attain perfect disinterestedness and fairness, but this does not mean that we should cease all efforts that can help promote disinterestedness and fairness. To infer that there is no important relationship between two things because neither necessarily entails the other is absurd. Anonymity helps to reduce all sorts of implicit biases. The fact that blind review is not a magic bullet that achieves perfect objectivity does not entitle you to dismiss the process as irrelevant.

    Reply

Designed with WordPress