Leiter Reports: A Philosophy Blog

News and views about philosophy, the academic profession, academic freedom, intellectual culture, and other topics. The world’s most popular philosophy blog, since 2003.

  1. Justin Fisher's avatar

    To be worth using, a detector needs not only (A) not get very many false positives, but also (B) get…

  2. Mark's avatar

    Everything you say is true, but what is the alternative? I don’t think people are advocating a return to in-class…

  3. Deirdre Anne's avatar
  4. Keith Douglas's avatar

    Cyber security professional here -reliably determining when a computational artifact (file, etc.) was created is *hard*. This is sorta why…

  5. sahpa's avatar

    Agreed with the other commentator. It is extremely unlikely that Pangram’s success is due to its cheating by reading metadata.

  6. Deirdre Anne's avatar
  7. Mark's avatar

Do journal editors have an obligation to disclose referee reports?

Brian Leiter

Philosopher Laura Schroeter (Melbourne) writes:

In September last year, I submitted a paper to Mind. In March, I discovered by chance that a senior colleague had written a referee report on the paper recommending conditional acceptance but had raised one main concern he thought should be addressed before publication. After many promptings, Mind finally made a decision to reject the paper this September. The editorial administrator attached comments from the editor (who is not a specialist in this area) and comments from one referee (which appear to be redacted). The comments from the colleague I’d spoken to earlier were not included; when I contacted him directly, he sent me a detailed 5pp report. Mind has not responded to an inquiry about whether they have a policy about not forwarding referee reports to authors. Moreover, the official rejection letter included the following proviso:

"Please note that referees' comments are supplied in confidence for the Mind editorial board.  They are forwarded to you as confidential reports, in order to provide you with feedback concerning your submission.  The Editor asks that you respect the confidentiality of this advice."

To be clear: I respect the right of editors to make the final call on a paper, and to override referees’ verdicts. But it seems to me that a policy of not forwarding reports is a real betrayal of the crucial role journals play in the discipline as intermediaries between authors and critics. When I spend time writing detailed feedback on an argument, I expect my report to be forwarded to the author – I wouldn’t go to all that trouble if it were simply advice to an editor. Personally, I won’t be refereeing for Mind in the future now that I know that they consider reports the confidential property of the editor alone.

I had not heard of a case like this previously; I thought referee reports, unless content-free, were always sent to authors.  What do readers think?  What experiences have they had?  If you name a particular journal, you'll need to sign your name to the comment; other comments must include a valid e-mail address at least.

Leave a Reply to Bill Wringe Cancel reply

Your email address will not be published. Required fields are marked *

50 responses to “Do journal editors have an obligation to disclose referee reports?”

  1. Clayton

    Had the exact same experience with Mind. In the first round I waited a year and received three referee reports. Verdict was R&R. In the second round, I waited another year and received three new referee reports. (I think I was told that my paper would go out to three new referees if I decided to resubmit.) In the second round, the first three seemed to be leaning towards R&R. If I recall, what was shown to me were three referees reports in which the referees said things that seemed to indicate that they were recommending R&R. (Tough to tell in light of the journal's policies about back channel communications with referees.) A fourth referee was called in that tipped the balance to reject. The journal wouldn't disclose anything about the substance of that report. This was frustrating because it took two years and the decisive reason for rejecting was kept confidential. The comments from the 6 referees were helpful, but they didn't identify the reason that kept the paper from being accepted. Would I say that there was an obligation to share that report or provide a summary? No, I suppose I can't say that. I can say that it would have been nice and it wouldn't have cost anyone anything. Maybe it's thought that it's easier to get referees to submit comments if there's the promise that their remarks would be kept confidential. I admit that I don't understand the wisdom of this possible policy, but maybe that's the thought. (I'd be less inclined to submit referee reports if I didn't think my work would be shared with authors.) Maybe it's thought that not offering reasons would decrease the kinds of complaints they'd receive when referees offer spurious reasons. At any rate, my guess is that there's an assumption on the part of authors that these reports will be shared in some form or other. Perhaps journals should be more upfront about what happens when you submit. (I haven't checked Mind's page to see if they say anything about the process. It's quite possible that they have been perfectly up front about this.) If nothing else, it would avoid causing the sort of distress that the practice seems to be causing.

    Reply
  2. Marcus Arvan

    Isn't there something also a bit dicey about a journal using a colleague as a paper referee? It's not at all uncommon, after all, for one's colleagues (particularly people who specialize in the same area) to know what one is working on, paper title and all (as it's not uncommon to seek feedback on working papers). Can't this practice be reasonably expected to compromise blind review? Doesn't it invite professional nepotism?

    Reply
  3. Paul Bloom

    I am the co-edtior of a journal, Behavioral and Brain Sciences, that often publishes philosophical work. What Laura Schroeter says makes perfect sense. Like every other editor I know, I almost always forward all of the reviews. For one thing, the reviewers provide comments and ideas (sometimes extensive) that are directed towards the author and might turn out to be helpful. As an editor, I can see why someone would be tempted to hold a review back — if I"m going to reject a paper, it is easier to do so if the author never sees a glowing review suggesting that it be accepted. But yielding to this temptation would be obviously unethical.

    The exception is when I get a very nasty review without substantive comment. I might hold it back, and just mention in my action letter that I got another review, but it contains no relevant content.

    Reply
  4. Rebecca Kukla

    Hm, I am the co-editor of the Kennedy Institute of Ethics Journal, and I admit that I have on occasion edited out bits of a referee's comments to an author. I have never done this to try to pull one over on the author though. Sometimes referees have said something that I found inappropriately insulting combined with clearly wrong. On one occasion a referee went on pompously about his own work and knowledge in a way that blew the anonymity of the process and came off as unprofessional. Things like that. I am open to being convinced by this thread that I owe my submitters the full reports, warts and all, but it has always seemed more helpful and professional to me to redact useless stuff of that sort.

    Reply
  5. ezio

    is this about

    1) authors wanting to see the reports to check whether the rejection was fair? in that case no way there is an obligation…

    2) authors wanting to see the reports to improve their paper? again, no obligation, journals are not there to improve our papers…

    3) referees wanting that their reports are seen by authors? here i agree with what has already been said, would be pretty annoying to have drawn up detailed comments just for the editor's eyes, but then again it does depend on the journal's advertised policy – which i must confessed to have never checked for any journal that i have refereed for

    Reply
  6. Matias Vernengo

    I had a similar situation with an economic journal. My co-author and I had sent the paper to a senior colleague, and he sent long detailed comments by e-mail, which were mostly positive and asking for further clarifications. When we received the rejection from the journal we got only excerpts from one of the referees, which were word by word from the e-mail we had received independently. The journal refused to send the other referee's report. I tend to agree that this undermines the role that a journal should have as a provider of unbiased critique.

    Reply
  7. Douglas Portmore

    In 2007, I refereed a paper for Mind. In the section of their refereeing form prefaced by "Comments on the paper for the author(s) (please ensure that your comments can be forwarded anonymously to the author(s) without the need for editorial intervention)," I included nearly two single-spaced pages of detailed comments on the paper. I later found out who had written the paper, and in discussion with one of the two authors I was surprised to learn that they had not received my comments. I later sent my comments to that author myself, who said that they were helpful as they were revising it for publication in Mind. I had recommended "accept without reservation." The paper is now published in Mind. I was annoyed to had been asked by the journal to provide comments for the author(s) only to find out that they had not been passed on to the author(s). I had assumed that this was an isolated incident, some sort of bureaucratic snafu.

    I think that journals should let referees know if they might not pass on comments to the author(s) so that we don't waste our time writing them in a way that will be useful for the author(s). I even point out typos. There's no point in doing that if the comments aren't going to be passed on to the author(s).

    Reply
  8. Brad

    I believe that as a referee my comments are, first and foremost, written for the editors of journals. I need to give editors adequate information in order to make a fair and informed decision about whether or not a manuscript should be published. But I do recognize that referees' reports are often shared with the authors of the manuscripts. Consequently, I write in a respectful manner, and with the thought that a person with feelings and ambitions wrote the paper and will see my comments. The sorts of comments I provide should be useful to the authors.

    Incidentally, a number of journals give referees the option of sending some comments to the author, and some to the editor only. I always opt for sending all comments to authors. This helps create the sort of transparency that should calm nervous would-be-authors.

    I think some would-be-authors are under the misguided impression that journals are a service for getting feedback on papers. Journals can have that effect. But that is not their principal function. Nor does a would-be-author have a right to be published in a journal.

    Reply
  9. doris

    I certainly would prefer to deal with journals who release to authors complete versions of all reports. I think journals also have an interest in following this practice.

    Withholding or heavily redacting referee reports raises issues about editorial transparency and impartiality, exactly the sort of thing that engenders all-too-familiar worries about cronyism and the like. Conversely, providing the reports goes some distance to addressing these concerns.

    The obvious reason for withholding is confideniality, but I expect that in the great majority of cases, no more than minor editorial redaction is required to ensure anonymity. (In addition, referees who wish to remain anonymous should be writing with that in view.)

    Hard for me to see, then, the reasons favoring anything other than full release.

    On a lighter note, I am happy to see that Mind has streamlined an editorial processes often criticized for being unacceptably long, and now only requires 6 referee reports.

    Reply
  10. www.facebook.com/profile.php?id=533113083

    Professor Schroeter raises a serious issue in a thoughtful way. And I strongly agree with her. I'm currently an Associate Editor of STUDIES IN HISTORY AND PHILOSOPHY OF SCIENCE — we have three such editors, and one editor in chief, Anjan Chakravartty — and our journal operates in the way that Schroeter indicates a journal should do. In my role, I commission referee reports — at least two — for each submission in my area, and then I make a judgment about the paper based on those reports (Chakravartty has the final say); but in all cases, we have the reports sent along to the author(s). I cannot see any reason to suppress referee reports, for they are the crucial means of providing authors with critical reactions to their work. In this particular case, one wonders what the editors at MIND were attempting to hide when they suppressed a report (and in this case, it seems that the report would have been quite informative for the author).

    This raises a larger concern that has been discussed on Leiter's site before: when will philosophy as a discipline decide that its mainline journals must all be run according to widely accepted best practices? We allow various kinds of deviations from those practices at major journals; as far as I can tell, the reason for these deviations is simply that the journals are not transparent. They are run by small groups (sometimes they are run from a single department) who presumably follow the norms set by their predecessors long ago. But long ago, as we all know, our discipline was not professional, and there were many sexist and racist practices that were widely accepted (many exist today, but when brought into the open light, they are much more difficult to defend now). It would be beneficial, in my view, if philosophers throughout the world who publish at least some work in English were to join together to demand that journals follow best practices. At a minimum, journals should commission blind referee reports and send them to authors in all cases. If the editors of a journal refuse, then we can follow Schroeter's lead and simply refuse to work for them anymore. As many philosophers have pointed out, almost all of our labor is free: we agree to referee papers because we care about the profession. Since our labor is not remunerated, we can remove it without any cost. No journal can operate without extensive cooperation from us. Junior faculty who must publish quickly to obtain tenure or promotion could submit their work to those journals — there are many of them — that have already adopted best practices. The answer might be: but MIND is a first-rate journal, so we surely cannot ask junior faculty to remove it from their radar. That is a good point. However, I might add that the ranking of journals in the profession is partially dependent on the number of submissions (and the related rate of acceptance), so if submissions to journals without best practices decline, and if submissions to journals with best practices increase, then the rankings will be affected. If a journal's reputation were on the line, its editors might decide to adopt best practices.

    Reply
  11. Lori Watson

    I was the Managing Editor at *Ethics* for two years in the late nineties, while John Deigh was Editor. Our referee forms had a section where the referees could indicate comments they wanted to remain confidential (just for the editors) and a section in which they made comments for the author. We never edited or redacted those comments, and always forwarded all referee reports to authors, along with any additional editor comments (depending on how far the paper got in the process, there may have been multiple Associate Editors offering comments). Finally, we also included a copy of all referee reports and editor decision letters to the referees, so the referees were informed of the final outcome and reasons. I think *Ethics* practices here constitute a model for best practices.

    Reply
  12. David Sobel

    I see only one legitimate ground for withholding referee comments from an author and it is the one Rebecca mentions above where the comments are unnecessarily insulting and/or obviously stupid. Withholding positive reasonable reports simply to make it easier to justify the editors verdict is shameful. This only helps make it seem that the editors are more justified in their decision than they in fact are, it is an abuse of the hard work of referees who reasonably assume that part of the point of their work it to help make the paper better, and it deprives the author of potentially useful feedback. I think editors should suck it up and allow that their decisions are not always easy to justify, but not misuse referees and block philosophical exchange in an effort to make it seem that their decisions are easier than they in fact are.

    Reply
  13. David Sobel

    I should have added that until Mind publically announces a change in this policy, I will be much less willing to referee for them or submit papers to them.

    Reply
  14. anon

    I recently had a paper rejected. In his email to me the editor indicated that he was attaching the referees' comments. Two sets of comments were included. I wrote back to him to ask whether those were all of the comments and he then indicated that there was in fact a third referee. The third referee had suggested the paper be accepted without revisions but the editor referred to that report as non-substantive- I presume that was why he did not forward it to me but I don't know. I think that if the editor is going to forward referee comments, then all sets of comments should be forwarded. At the very least it would have been encouraging to this author to have seen the third report.

    Reply
  15. David Sobel

    I suspect Laura meant the referee was a colleague in philosophy, not in her dept.

    Reply
  16. Anon

    Often I have received rejections (from many different journals) with referee comments that were *obviously* edited. Many times, the "referee report" would be reduced to one or two lines.

    I just assumed that this was standard practice, since (understandably) editors do not want to provide the author with ammunition (i.e., positive comments) to contest the editor's rejection. Editors I assume have to make lots of tough decisions that *are* contestable (though hopefully not entirely arbitrary). But such is the reality. If *uncontestable* rejection was the condition for rejection, then journals would be obliged to accept way too many papers.

    Yet if negative referee comments are being withheld, I don't see the rationale in that (unless the criticisms are weak or insulting or such).

    Reply
  17. VL

    Coming from the biomedical sciences, I must say it is surprising to me that any journal would not send the full referee reports–if that happened to a scientist, there would be outrage, angry phone calls to the editors, and a public shaming of the journal. Take, for example, this October 5, 2012 post here about Biology Letters, a journal of the Royal Society: http://svpow.com/2012/10/05/we-will-no-longer-provide-peer-reviews-for-royal-society-journals-until-they-adopt-honest-editorial-policies/ (The entry is worth a read for insight into possible reasons journals might reject papers recommended by reviewers: in short, it raises their rejection rate, and thus their perceived status.)

    There are two purposes for reviews: to help editors decide whether to publish a manuscript, and to improve the manuscript in the process. Even if one sees referees merely as gatekeepers (emphasis on purpose #1), raising the quality of publications in general (purpose #2) is certainly part of a thick conception of gate-keeping. Furthermore, reviewing manuscripts is a skill in itself, no less than writing papers in the first place, or writing grants or giving talks; I view all of these as essential skills for scholars. A quick google search just now has turned up a number of sites with good suggestions for reviewing manuscripts, but I think the best discussion comes from a panel of editors who talk about what makes for a good review:
    http://orgtheory.wordpress.com/2011/05/31/the-editors-speak-what-makes-a-good-review/#more-15599

    Reply
  18. Rebecca Kukla

    So several people seem to agree, by implicature, that withholding obviously stupid and/or insulting negative comments is ok. Is that what people think? I am planning on taking y'all's opinion very seriously when it comes to my own practices. I agree that withholding positive comments or potentially substantively helpful comments of any kind is not ok.

    Reply
  19. Douglas Portmore

    I agree.

    Reply
  20. VL

    To Rebecca: Taking on the responsibility of redacting unhelpful comments in reviews might spare someone's feelings, but it has two negative aspects: it saps your time and emotional energy, and it does not address the root of the problem, which is that the reviewer has probably never been instructed in how to write a good review, as opposed to simply critiquing a manuscript. I'm not sure what should be done; it probably would be irksome to an offending reviewer to be told how to write a good review after the fact, but perhaps you could lay out what you/your journal seeks in a good review, and include that with the initial request to review a manuscript. This way prospective reviewers are subtly inculcated with a nobler set of standards right from the start, and you then have grounds to go back to them if they violate those standards egregiously in their reviews. I view this as a slow process of cultural change/individual education; I also train graduate students and postdocs in the art of writing helpful reviews, and know faculty who incorporate this training into their programs.

    In the meanwhile, I must say there are many reviews I wish I hadn't seen, but it's part of the rough and tumble of science. One learns to tamp down one's defensive response to unwarranted aggression, to take the high road, and move forward. Learning how to deal with bullies is, alas, another aspect of succeeding in academia. (In the sciences, it seems no journal editor holds back or redacts a vicious review. Scientific editors evidently can be an unfeeling lot.)

    Reply
  21. Sam Rickless

    Although editors are probably not *legally* required to provide prospective authors with complete, unredacted feedback from reviewers, they are, it seems to me, *morally* required to provide to prospective authors whatever potentially helpful information they received from their referees (assuming that this information was not expressly designated for the editors only). Why? For one thing, it shows prospective authors that the decision (positive or negative) was not arbitrary or unjustified. For another thing, it has the potential to do a great deal of good, because comments from reviewers, whether positive or negative, (as others have already pointed out) can be really useful to prospective authors. I agree that it is not the *function* of a journal to provide comments to authors: the function of a journal is to publish the best papers it receives. But if an editor has received comments that can be shared with an author without compromising blind review, then s/he should do so. There seems no good reason not to.

    Rebecca mentions a type of case in which a referee says inappropriately insulting and/or clearly wrong things. Rebecca and David (understandably) suggest that in this sort of case, the insulting/wrongheaded comments should be redacted before they are sent on to the author. My own reaction is different. It seems to me that if a referee says inappropriately insulting and/or clearly wrong things, then that referee's report should not be trusted, the report should be thrown out, and a new report from a different referee commissioned (if there is any chance of acceptance or revise-and-resubmit). This may prolong the process, but OTBE it is better that the process be fair than that it be short. Moreover, it seems to me that editors have an obligation (to the profession) to point out to referees who are abusing the refereeing process (by saying insulting and/or clearly wrong things) that such reports are unacceptable.

    Reply
  22. Steven French

    I'm co-editor in chief of the British Journal for the Philosophy of Science and I agree with Brad: referees' reports are written primarily to help the editor(s) come to a decision. If the comments may help the author improve the paper then of course they should be forwarded but there are many reasons why they might not or why they might be edited. As for the suggestion from my colleague over at Studies that all journals should conform to some (centrally imposed) set of rules or guidelines, leaving aside the practical issues (having to do with different publishers, different society 'owners' etc), I don't quite see why appropriate diversity can't be aligned with good practice in terms of anonymity, courtesy etc.

    Reply
  23. Jonathan D. Jacobs

    I'm also interested in thoughts about Rebecca Kukla's idea of withholding obviously stupid and/or insulting negative comments. I've done this myself, but if there are good reasons to send the comments along, perhaps with an editor's note, I'd be interested in hearing them. (I'm the new editor of The Modern Schoolman, to be renamed Res Philosophica with the 2013 volume.)

    Reply
  24. Jamie Dreier

    I do not remove inappropriately insulting remarks from referee reports when they go to authors. In principle I can imagine remarks that were so awful and gratuitous that I'd feel I had to take them out, but that has never happened in the four years I've been an associate editor (for ETHICS). I do not agree with Sam Rickless that a report containing inappropriately insulting remarks shows that the report should not be trusted; many gratuitously insulting philosophers have sound philosophical judgment. But if the report has several stupid (as Rebecca Kukla puts it) comments, I agree with Sam Rickless that the right course of action is to find another referee.

    Anon at #16 suggests that the reason for withholding some (positive) reports is that "editors do not want to provide the author with ammunition (i.e., positive comments) to contest the editor's rejection." I have never had an author contest a rejection. I think it is more likely for an author to contest a rejection on the ground that a negative report was very badly mistaken or confused; I have not as an editor had an author contest on those grounds, but I do know of a few related cases.

    Finally, I would like to know who Brad at #8 is, because I think everything he said is exactly right and I want to send him some papers to referee. (I guess that's a good reason for him to maintain anonymity.)

    Reply
  25. Thom Brooks

    I'm the editor and founder of the JOURNAL OF MORAL PHILOSOPHY which I've run for about 10 years. I am shocked to hear of referee reports not forwarded to authors. Referees provide an *invaluable* service to the profession: I try to make a point of thanking past referees in person where I can. To think this service might come to so little genuinely beggars belief.

    I agree that there might be times where some redaction is necessary. I would think such cases very rare. In my case, I would – in the 2-3 cases I can recall – warn the authors that the reports were somewhat inappropriate. In every case, I've been thanked for my frankness.

    But again – when colleagues are asked for their comments (and asked…and asked), it must be expected these comments will be communicated to authors. I'm shocked by the stories above.

    Reply
  26. Paul Bloom

    I've sometimes given feedback to reviewers, particularly those early in their careers, about how to write a better review. (Sometimes their reviews are too damn long, and get caught up in irrelevant detail.) But actually, at least in my own experience, the nastiest and laziest and most dismissive reviews are from senior people, often major figures in their field. They've certainly know what a helpful review looks like; they just choose not to write one.

    Off-topic, but I can't resist adding that in my ten or so years editing BBS, my experience is that philosophers tend to be the most thoughtful and constructive reviewers.

    Reply
  27. Sam Rickless

    Jamie (at #24) is right that there are gratuitously insulting philosophers with sound philosophical judgment. But as an editor I would worry about whether the gratuitous insults in any particular case are the product of (or likely to be correlated with) bias or prejudice or some other serious flaw that casts doubt on the soundness of the referee's judgment. In cases in which the editor is an expert in the relevant sub-field and is able to judge that the insulting referee's comments are really very good despite the insults, then redaction seems appropriate, in keeping with Rebecca and David's suggestions. But in cases in which the editor is *not* in a good position to judge whether the insulting referee's comments are really very good (e.g., the editor lacks relevant background knowledge of the relevant subfield), then it still seems to me that the insulting comments should be thrown out. I don't want to go so far as to say that gratuitous insults should always be treated as completely irrelevant to the reviewing process, that they should be ignored or simply redacted.

    Reply
  28. Junior Tenured Philosopher

    I recently had a paper rejected by Mind, and found the process itself frustrating. It took two months just for *an acknowledgement of receipt of the paper*. After 8 months, a decision came, but only after I wrote the editor to check for signs of life. In addition to the boilerplate language quoted by Laura Schroeter, the editor's response also included this: "Two referees returned advice on the paper; one has supplied comments-for-author. These are copied below." What is (possibly intentionally) omitted from this are things like (1) whether the other referee supplied comments to the editor, albeit not "comments-for-author", or none at all (2) what the ultimate recommendation of this other referee was (e.g., whether the other referee advised publication, and whose advice was not heeded by the editor), (3) whether there were additional referees who did not return comments and/or a recommendation at all. It seems to me authors have an interest in knowing at least some of these things.

    Reply
  29. Allen Wood

    As a sometime editor and as a continuing referee of papers, I know that sometimes the referee does not want his or her comments relayed to the author. I believe journals must respect the wishes of referees in such cases. This seems to me a legitimate reason for not forwarding them.

    Reply
  30. Jamie Dreier

    Quite right — my experience is limited in that, as an *associate* editor, I handle only papers in my own sub-field. But I can see that as sole editor you'll have to deal with papers you cannot judge on your own.

    Reply
  31. Sam Rickless

    Hey Allen, I love you man, but what if all referees did that?

    Half-joking aside, what would be a good reason for a referee not to share her/his comments with the author?

    Reply
  32. Anon Junior Philosopher

    Adding to the chorus here: When referees write reports, I presume that they aren't doing it as a favour specifically for the journal in question, but instead are trying to do something for the profession as a whole. That means helping the editor make their decision, but also helping the author improve their paper, and also hopefully thereby helping other journals where the paper might be sent in the future. These wider goals aren't served if referees comments aren't made available to the author. So withholding referee comments seems to me to be unfair to the author, unfair to other journals who may later receive the paper, and unfair to referees. This is all assuming, of course, that the report is substantive rather than insulting, and is not marked as being for the editor only.

    Finally, just to point out something which I hope everyone knows but hasn't been explicitly said: even if an editor rejects a paper, a positive referee report (even one with no substantive comments) provides helpful signals to the author about what to do with the paper next. So it is not as though withholding such reports is costless.

    Reply
  33. Steven Hales

    I'm refereeing a submission to Mind right now. This discussion is making me doubt the value of writing a very detailed report. I have a hard time believing that other referees will not have the same opinion of the submission that I do; it is an obvious case. So an in-depth report will be of no value to the editors who presumably just need another thumbs-up or thumbs-down. Therefore, such a report is of value only to the author. But if the author won't be sent the report…

    Reply
  34. Rebecca Kukla

    When I receive a referee report that just seems glib and wrong-headed I di indeed solicit another report, unless the other report I already have makes a completely convincing case for rejection. But getting a third report after the first two are in is far from cost-free, as the process is already frustratingly slow for authors. I don't find it that rare that a report seems helpful to me (helpfully positive or helpfully negative) but includes insulting, inappropriate, or just clearly ill-thought-out moments. I am surprised that others seem to find that rare!

    Anyhow none of that is to defend my redaction policy – just to clarify it. I do indeed fairly often move onto a new reviewer in the face of a referee whose report I don't trust.

    Reply
  35. Donald Bruckner

    I've always approached my job as a referee assuming that my comments would be passed on to authors and weren't only for the editor's benefit in making a decision. If I thought that it was just the latter, I'd spend *much* less time writing a careful referee report, and would simply give a recommendation to the editor with a very brief justification. I'm thinking I've sometimes been asked to do something like that, but otherwise I assume my comments will go to the author and I approach the task under that assumption.

    So given that it's unlikely that all requests from handling editors will be accompanied by a clear statement as to whether the report will be forwarded to the author (or that all journal websites will say), what do people think that people asked to referee should do? I'm inclined from now on to reply to the handling editor to ask if my report will be sent to the author. The trouble with that is that it would substantially increase what must be the already huge e-mail volume that managing editors have to contend with.

    Reply
  36. Tom Hurka

    1) Just wondering: In how many threads about bad journal practices does MIND play a starring role? Just about all of them?

    2) From my journal-editing past: Often negative comments in a particular referee's report didn't play a role in my decision not to publish a paper — I thought they were mistaken but thought there were other problems with the paper, perhaps highlighted in other reports. Just sending the report might therefore give a false impression of why the paper was rejected. An accompanying note explaining the reasons for rejection is therefore helpful, but is extra work for the editor.

    Reply
  37. Thom Brooks

    It might be useful to note a link to my paper on guidelines for referees – http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1719043

    Reply
  38. David Velleman

    As an editor who has recently been accused of moral turpitude on this blog, I would like to dissent from the fundamental presupposition of this thread, that journal editors have special moral obligations to authors. There are disciplines in which the membership fees charged by the professional organization pay for subscriptions to "official" journals. It seems to me that those journals do indeed have special obligations, because subscribing to them is required for participation in the profession. Since philosophy is not such a discipline, I see no basis for the sort of obligation under discussion here.

    Authors should of course ask editors about their editorial policies, and editors are under the general moral obligation not to lie. Authors should share their experiences with various journals. A wiki for this purpose would be a great idea. (Maybe one already exists.) The APA should compile and publish a list of journals and their editorial policies or refusal to disclose them. (There used to be such a list. I don't know whether it still exists.) Hiring, tenure, and promotion committees should familiarize themselves with that list, and judge publications accordingly. But if someone wants to publish a journal without a policy of blind refereeing, they are fully entitled to do so. For that matter, if Jane Coe and John Doe want to publish the Jane Coe and John Doe Journal of Philosophy, and if Jane Coe and John Doe have a good eye for interesting philosophical work and a distinctive sensibility, then I say: More power to them. I won't submit my work to the JC&JDJP. I and my colleagues — and, I would hope, hiring, tenure, and promotion committees — will judge the content of the JC&JDJP differently from that of Philosophers' Imprint, which has a strict policy of blind refereeing. But maybe some readers and some authors will find the JC&JCJP especially interesting or especially hospitable to their own sensibility. Good for them.

    A journal is not a public utility. Every journal is indeed a "gatekeeper" to professional opportunities, but it is also a *gate*, providing an additional venue for philosophical work. If the JC&JDJP didn't exist, there would be one fewer avenue by which to disseminate and read philosophical work. All of this talk about Jane and John's special moral obligations is just so much empty moralizing.

    Pile on.

    Reply
  39. Bill Wringe

    The most salient obligations in this case seem to be to the referee who has spent time giving constructive comments on a paper, in the expectation that they will be passed on. David Velleman's fulminations about John and Jane Doe completely ignore this.

    Incidentally, why do so many editors say they 'don't have time' to pass on comments? Surely that's something that could be done automatically (at least by those journals that use automated submission software). Redacting them is another matter, of course, but I'd be willing to endure the usual measure of passive aggressive sarcasm for the privilege of finding out the kind of basis on which Nous or Ethics (say) are rejecting my stuff.

    Reply
  40. A Junior Philosopher

    In suggesting that editors have no 'special moral obligations' to authors, David Velleman ignores the entrenched institutional role that some journals have acquired. Velleman of all people should know that giving someone a new option does not necessarily leave them in the same place as before should they refuse it (cf. his published argument against assisted suicide). Junior members of the profession cannot, without consequence, refuse to submit to prestigious journals. Do expectations of turnaround times measured in weeks or months not years; of forwarding of referee feedback to authors; or of a triple blind review process unreasonably burden editors? Even though other editors with equal or lesser resources manage to accomplish all of these?

    Moreover, what about implied obligations to *referees*? The MIND referee report form I was sent this year (very sensibly) has two sections for comments, described as follows:

    'Comments on the paper for the author(s) (please ensure that your comments can be forwarded anonymously to the author(s) without the need for editorial intervention)'

    AND

    'Further comments for the editor (these comments will not be passed on to the author(s))'

    On reading this thread, I must wonder if my time spent conscientiously filling in the first section was precious time wasted, and wasted for no good reason at all.

    Reply
  41. David Sobel

    I see the main question before us as whether, given that most journals do not currently speak to the question of whether they pass on almost all reasonable referee reports (or the parts intended for the author) to authors, and given my sense that referees assume that such a practice is the norm unless told otherwise, it would be best under current conditions for editors to have a policy of passing along referee reports except under quite unusual circumstances. I don’t especially feel like saying that editors have a special moral obligation to authors but I do think that what is going on in the cases brought to light here is not cool and the only rationale I see for such practices in the cases before us are lame. I agree that a journal could clearly state different policies, and then if people wanted to submit to such a journal under such an understanding I agree with Velleman, more power to em. But that situation is not the situation we are in and it is not relevant to evaluating the cases that are being brought to light here.

    Reply
  42. Neil Levy

    David Velleman is of course right that journals may adopt all sorts of policies entirely ethically. In the actual world, though, most of us are under the impression that journals trade speed for detail. When we referee for Mind, we know that the author expects to wait a long time. We try to compensate them, a little, by our detailed reports. Given its turnaround times, Mind of all journals does have an obligation to pass on reports. It can waive that obligation by making it explicit that reports may not be passes on. If it does that, it might prove its turnaround times. At the moment, it seems to give us the worst of both worlds.

    Reply
  43. ben wolfson

    Where did non-blind refereeing come from, in Velleman's comment? Have people been talking about non-blind refereeing thus far in this thread? (Not that I can see.) So (as far as I can tell) it's tangential to wonder why, if it's really true that Jane Coe and John Doe have a good eye for distinctive philosophical work and an interesting sensibility—something I'd hope for editors and referees at journals that do employ blind refereeing as well—anyone would judge the content of the JC&JDJP differently from the content of Phil Imprint. How are such judgments supposed to go, anyway? "Yes, the arguments are forceful and innovative, the command of the literature impressive, etc. etc. etc., BUT the journal doesn't employ blind review, you know, so the paper must not be very good after all." Or perhaps the judgment is supposed to be being made by someone who can't evaluate the paper just by reading it? But I would hope any such person would be unwilling to judge that a paper in, say, Mind must be good, having, after all, been published in Mind.

    I would also like to note the rhetorical move of denying that journal authors have *special* *moral* obligations to authors. No one had previously mentioned moral obligations, let alone special moral obligations. Is that what's in question? To establish that there's a special moral obligation born to authors by editors seems prima facie more difficult than to establish that there's, I don't know, a perfectly ordinary professional obligation.

    Reply
  44. Jamie Dreier

    Just to correct an impression left by Bill Wringe's comment: ETHICS always passes along to authors the reports submitted by referees. (I believe Nous does, too, but I can't vouch for it.)

    Referees are able, if they choose, to add notes to editors only. Few make use of this option, and exceptionally few add more than a sentence or two.

    Reply
  45. Dan Dennis

    David Velleman’s comments are based on a case where someone starts and runs a journal as their own private idiosyncratic fiefdom. Perhaps he is right in what he says about such a case. However it is irrelevant to this thread. In this thread we not interested in journals of this type. Rather we look at what should be the practices of reputable, established journals.

    In nearly all cases these were not started by the editors who currently run them (Mind started in 1876). Rather the editor is merely the custodian of an institution. It is reasonable to think that those who started and contributed to the flourishing of the journals did so as a service to philosophy. The journals were built up to serve philosophy. If someone takes on the role of custodian of an institution devoted to service to philosophy, then he should be focused upon service to philosophy. His practice as editor should reflect that. Clearly passing on referees comments to authors is a service to philosophy (at very little cost) so he should do it. It would be practically irrational to be devoted to service to philosophy and yet not do something which is of service to philosophy and comes at such little cost.

    Someone who is currently an editor of a journal out of self-interest (e.g. prestige) may not act in the interests of the profession (e.g. not pass on referees’ comments). He is in conflict with his duty as custodian of this valuable institution, so should resign his job. If he will not, and if there is a board of trustees with overall control of the journal, then they should replace him with someone else. This is also the case in the (hopefully more common) situation where an editor who has the best of intentions finds that in practice he does not have the administrative skills, or the requisite time, to do as good a job as he should.

    Reply
  46. Bill Wringe

    @Jamie Dreier

    I should apologise for what I said about Ethics: I was misremembering the gist of a recent rejection, which told me that although the editor had selected my paper for anonymous review, they were unable to provide me with comments.(What I had misremembered was that the anonymous review was done by one of the journal's associate editors, who was writing to me directly, and not by an external referee. There's a reason why I misremembered that – namely that its slightly unusual that in an anonymous review situation the reviewer and the individual reviewed will each know the other's identity. But I was nevertheless wrong.)

    As far as I can tell, Nous is another matter – here's the boilerplate language from a recent rejection:

    'The editorial workload at present makes it impossible to provide you with detailed comments. Given the sustained surge of submissions after the end of our moratorium, we cannot provide comments on every rejected paper. We focus rather on arriving at a well-informed judgment without undue delay.' The email goes on to talk about the fact that referees may have submitted reports that don't show up on the journal's software system.

    That certainly suggests (it suggested to me, and I find it hard to believe that this was accidental) that the paper has been sent out for review, but the editors haven't got time to pass on the comments. If what it means is that the paper hasn't been sent out for review,then I think the journal should be upfront and say so.

    Few journals are upfront about this; and some journals will not even say, when asked directly, whether a paper has been sent out for external review. Given the editorial practices of Mind reported in this thread, we certainly can't infer from the fact that no referee's reports are sent to the author, that the paper was not sent out for external review.

    Reply
  47. David Velleman

    @Ben Wolfson: Yes, some earlier commenters referred to moral obligations (e.g., Sam Rickless #21).

    The purpose of my post was to point out that philosophers rely on unexamined assumptions about the nature of the journals in our profession. Philosophy journals are governed and managed in many different ways, by many different bodies and individuals. Some of the journals that you regard as venerable institutions may in fact be run as the fiefdoms of their current editors — which is not incompatible with their being devoted to the service of the profession. (The idea that anyone edits a journal out of self-interest is laughable.) My example of the JC&JDJP was meant to be an extreme case, but it is at the fictional extreme of a very real continuum, of which the other extreme is represented by MIND. Journals are not, as I put it, public utilities regulated by uniform standards. Nor should they be.

    Technology has democratized academic publishing. From that perspective, the Mind Association can look like a bastion of an old order, defending the vested interests of a privileged few. Ask yourself: how are editors chosen, and how do they choose referees? The whole system is based on academic pedigree (a system of which I myself have been a beneficiary, thanks in part to the blind luck of where my parents could afford to raise me, where they could afford to send me to college, and so on).

    Today we have online archives, self-published works-in-progress, scholarly blogs, and so on. What we should be thinking about is not how to shore up a peer-review system that was developed for a now-outmoded model for disseminating scholarship but rather how we can design new systems for the way that scholarship is already being disseminated in the real world. We need to be thinking up new systems of peer review, so that readers can have reliable guides to the best of the online archives, personal websites, blog posts, interviews, etc. We need to be thinking about the problem of permanence in a world of endlessly revisable publications. We need to be thinking about how to redesign our credentialing practices for a world in which the best scholarship may be identified in as-yet unimagined ways.

    There are scholars who are working hard on these problems, but no one has yet come up with a compelling solutions. In any case, it is not productive to spend our time wagging our fingers at journals that don't conform to the practices of an institution founded in 1876.

    Reply
  48. Tim O'Keefe

    David Velleman @47 says, "There are scholars who are working hard on these problems, but no one has yet come up with a compelling solutions. In any case, it is not productive to spend our time wagging our fingers at journals that don't conform to the practices of an institution founded in 1876"

    But we're not here talking about the overall practice of blind peer-review, but a narrower question of whether–given that they are using blind peer review–journal editors ought to disclose referee reports. There is currently a default social norm within the philosophical community that they ought to do so, and I think that this is a useful norm. Since it's merely a social norm, though, it's going to be upheld by people doing things like wagging their fingers at folks who deviate from the norm. That seems useful to me.

    Reply
  49. David Velleman

    I wasn't responding to the manifest topic of the thread; I was responding to a presupposition behind it, namely, that there are practices to which all journals should conform. This view is made explicit by commenter #10, who asks, "[W]hen will philosophy as a discipline decide that its mainline journals must all be run according to widely accepted best practices?" Even when not so frankly stated, this view is presupposed throughout the thread, which is a discussion of how journals (presumably, all journals) ought to treat the comments of reviewers.

    Obviously, editors should not mislead referees as to how their comments will be used, and they should not mislead authors as to their editorial practices. Beyond that, in my view, discussion of how journals ought to be run — and especially how editors are "morally obligated" to run them — is misguided. The referees for Mind almost certainly believed, with justification, that their comments would be conveyed to the author; the editors almost certainly knew that their referees had that justified expectation. The editors therefore did something wrong — not by some "best practices" for journal management but by simple standards of candor. Provided that editors follow those and similar rules of good behavior, then they may run their journals by whatever practices they like — the more various, the better.

    My own view, expressed in comment #47, is that the entire system of peer-reviewed publication is outmoded and utterly broken. Arguing about standards that shouldn't exist, for a system that no longer works, simply distracts from the real issues.

    Reply
  50. Daniel A. Kaufman

    This strikes me as one of these conversations–common among philosophers, it would seem–where things become much more complicated than they need to be or than they deserve.

    As a teacher, I do not take points off of a student's test or paper, without explaining the reason for the demerit. Why, then, as a referee or editor, would I reject a person's submission, without explaining to them why?

    Is this a matter of moral duty and obligation? Probably not. But why does it need to be? There are indefinitely many shades of decency and indecency that come before we get to that sort of baseline point. Invoking morality in this context is like invoking legality, when all the person wants to know is why you're being a jerk, rather than a nice guy.

    Beyond the question of decency, there is also the point of the purpose of the endeavor in question. Testing students is part of a larger learning process, and consequently, should be conducive to that process. Explaining point reductions is conducive in this way, while failing to do so is not. Publishing articles in philosophy journals is part of a larger process of expanding human knowledge. Explaining rejections is conducive to that purpose, while failing to do so is not.

    As for the "workload" argument….please, let's be serious. Undoubtedly, we all work hard…and I would argue that those of us at teaching-heavy institutions, without graduate assistants, work a lot harder than people in the posh places. But none of us is working all that hard. We're not digging ditches. We're not pulling 18 hour shifts in emergency rooms. We're not driving trucks cross country in sleet and ice. And we get three months a year off. Surely, there's plenty of time to type up a few comments, explaining why you rejected someone's paper.

    You know…for decency's sake.

    Daniel A. Kaufman (Missouri State)

    Reply

Designed with WordPress