Leiter Reports: A Philosophy Blog

News and views about philosophy, the academic profession, academic freedom, intellectual culture, and other topics. The world’s most popular philosophy blog, since 2003.

  1. jason Yackee's avatar
  2. a bit off topic's avatar
  3. s. wallerstein's avatar
  4. David Wallace's avatar
  5. Daniel Kaufman's avatar
  6. Daniel Kaufman's avatar
  7. Daniel Kaufman's avatar

Reader experiences with peer review?

Brian Leiter

A reader writes:

How do people feel about peer review? Does it really help? Has it made papers better in authors’ experience? There is an overwhelming amount of commentary about how peer review is no longer in any sense the guarantee of quality that it is often assumed to be. The publishers have not wanted to take this message on board – much like they didn’t want to take open access seriously – but within the academy, I’m seeing critiques from all disciplines, and particularly from the sciences, which drove the uptake of peer review for the social sciences and humanities.

In my view, the fundamental problem is the way that the modern university fetishizes “output” in terms of journal publication (and books depending on the discipline), which creates enough perverse incentives that lead to mediocre writing, plagiarism and outright fraudulence, which to be fair, can’t be detected in most cases by peer reviewers.

Here is a good and representative summary of the various discontents.

 https://experimentalhistory.substack.com/p/the-rise-and-fall-of-peer-review

The main pernicious effect of peer review in philosophy (but probably in other disciplines) is to encourage conformity and conservativism.  That may be unavoidable, and built into the idea of a Wissenschaft.

Thoughts from readers on peer review?   Submit your comment only once, it may take awhile to appear.

Leave a Reply

Your email address will not be published. Required fields are marked *

14 responses to “Reader experiences with peer review?”

  1. Phil

    I remember the three-line review devoted entirely to complaining that I hadn't referenced the most basic and fundamental texts in the field, written by an author who I had in fact read but not cited (I know because, whether from arrogance or carelessness, the review was signed); I remember my own (rather more detailed) review of a paper which struck me as so wrongheaded as to be almost physically painful to read, but which was published nevertheless because neither Reviewer 1 nor the editor felt as strongly. It's a flawed system – although, of course, it does seem to work a lot of the time.

    Coming to academia relatively late in life, I approached research with the naive assumption that one wrote in order to tell a story or else to set out one's ideas: the idea of a literature review was new to me, not to mention alien. Twenty years later I still find it odd to devote the first half of a new paper to gratefully acknowledging the contributions of Drs A and B, saying "yes, but then again no" to C and "no, but then again maybe" to D, and shaking my head over the lamentable errors of E and F – but you better had if you don't want to risk a three-line rejection from Dr D (or Dr F, who would far rather be attacked than ignored). Perhaps this system or something like it is "built into the idea of a Wissenschaft", but it's an awfully cumbersome way of presenting anything new (and we only publish when we're saying something new, right?).

    Reply
  2. CallMeACynic

    My worst experiences with peer review have occurred when reviewers reject the paper because they didn't like the conclusion or because they were committed to the view which was being criticised. In these cases, they found some small quibble which they could blow all out of proportion to justify rejection. So one problem with peer review is that it can be used to prevent philosophical debate from taking place.

    My next-to-worst experiences with peer review have occurred when it was clear that the referee just didn't understand the paper. One time I received a report from a top journal which was so incoherent and ungrammatical I wondered how the editor could have sent it out without being ashamed.

    At least philosophy doesn't have exorbitant submission fees, like some journals in the sciences. One of my friends had to pay a $500 submission fee only to have the paper get desk-rejected with no explanation in 48 hours. And they didn't refund the submission fee.

    Reply
  3. toro toro

    I've had some awful experiences with peer review, like anybody who's been at this long enough. But I also know that it makes my work vastly, vastly better than it otherwise would be, even in the rare cases where the referees at the first place I submit it wave it through with no or minimal corrections. Preparing to face that gauntlet is an absolutely essential part of quality control.

    Reply
  4. Tom Cochrane

    I think most peer reviewers are pretty conscientious and I've benefitted from 90% of them. I feel far more confident about a paper that has passed through the rigours of peer review than one that hasn't. I just wish reviewers would remember to put themselves in the shoes of the person receiving the report and remember a couple of simple principles:
    – If you're going to say that literature hasn't been cited, say what that literature is, and preferably why it is relevant, otherwise your comment is bullshit.
    – A 1000 word review is often as bad as a 3 line one. Minimise your pontifications.

    Reply
  5. Mohan Matthen

    My experiences with peer review have sometimes been irritating, but I am currently having a round of R&Rs where one reviewer has really improved my paper and made me understand my own topic better. I won't say anything substantive that would reveal my identity to that reviewer now, because s/he might be reading this. But let me say that on first submission, Ref provided a beautifully economical dissection of really important and central points, all the while maintaining a courteous "Here's something I don't quite follow in this otherwise nice paper." That resulted in major improvements, but when I resubmitted, Ref came back with "Oh that was good, but there are still two points I am a little puzzled about." The first of those two "small" points was a revelation! I am just so grateful. I am a senior person past the end of his career (which, it seems, was not distinguished enough to have his retirement conference noted by Brian–there I managed to get that in!) So, I didn't expect to have a wonderfully constructive grad student/supervisor experience again. Thanks Referee 1! You've taught me a lot.

    Reply
  6. Michel

    From the writer's side, there's no question that my work is improved by the peer review process. And the proof of that is in the pudding: the revised versions of my R&Red papers are significantly different, and I'm much happier with them.

    From the referee side of things, however, I often feel like the changes made in response to verdicts of R&R are rather cursory, and closer to what I'd expect for an acceptance with minor revisions.

    Reply
  7. grad student

    In my experience, some referees have summarized their reason for rejection with "I just wasn't persuaded by the arguments the author offered". If being persuaded is the standard for peer review in philosophy, then we're screwed: when was the last time any of us read a paper or attended a conference talk for a view we disagreed with and walked away persuaded?

    Reply
  8. VBBaltzly

    Commenters may wish to consider the distinction between "peer review" and "blind review" more specifically. Bob Goodin's judicious gate-keeping at the Journal of Political Philosophy, e.g., is an instance of the former, but not the latter.

    I've read (though I've never attempted to independently verify this claim) that–at least up until WWII (if not somewhat beyond)–most peer review took the former shape: the well-known editors of the journals performed the peer review in non-anonymous fashion.

    Do many of the current issues with peer review arise from the lack of accountability that's perhaps inherent in a blind-review-based system?

    Reply
  9. Steven Hales

    Abolishing peer review won't matter to Adam Mastroianni, who has a Harvard PhD and co-authors papers with Dan Gilbert in Nature. He'll get his ideas read and accumulate plenty of professional capital anyway. But for someone with a no-name PhD and no big connections, getting past peer review into a major journal is the only realistic way to get ahead. Getting rid of peer review will not eliminate journal prestige, but it will mean that those journals will be exclusively filled with big shots, the well-connected and credentialed, and hot new scholars with plenty of buzz. Is that the future we want? I'm all for improving peer review, but let's not pretend it accomplishes nothing whatsoever.

    Reply
  10. Grad Student

    As a junior scholar, my experience with peer review so far is to wait forever. No matter where I submit, it seems to take them a year to actually collect reviews. Usually so they can reject the paper because one of the reviews was lukewarm.

    I have a paper under review right now, and in the time it's been out for review, I've proposed a special issue to another journal, recieved the submissions, organized a full peer-review, and had the accepted submissions copy edited. I' almost certain it'll be published before my paper is reviewed. Since the top journals have no shortage of papers, it feels like they can just screw you around forever with no consequences, except to your career.

    Reply
  11. Fool

    If peer review makes fields too conservative (which it does), perhaps it should be counterbalanced and combined with a factor that has been reliably shown help academic fields evolve in better new directions – https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6814193/

    As long as we maintain peer review, it seems, we should also institute compulsory euthanasia for scholars who reach a certain citation count.

    Best of both worlds. The Synthesis.

    Reply
  12. Thomas Osborne

    I have had many very positive experiences with peer review. Sometimes the reviewers have caught straightforward blunders. More often the reviewers pointed out where there was unclarity, either directly or indirectly through misunderstanding what I was saying. In my experience readers often misunderstand what they are reading, so it is quite helpful to have someone point out potential misunderstandings. It seems to me that the anonymity allows the reviewers to be more direct than they might be at a conference. I have also had many experiences where the reviewers didn't bother properly reading the text, took too long, said things that were obviously wrong, etc. But it is probably worth it to have the good impact. And usually there is something helpful that you can take from even a review that is poorly done. I should say that at least once I have made changes due to a review and then had people tell me that the paper was better before I made the changes.

    The reviewers often reflect the mindset of the potential readers, so even if the reviews are narrow or have a hard time understanding something new, it is helpful to have the feedback. It seems to me worthless to publish if nobody reads you or understands what you are saying.

    Reply
  13. Disgruntled Continental

    In the past two years, I have sent some things to Continental Philosophy Review only for the editors to discuss my work within a day or two and telling me No. It's so bizarre there that I know my work is not being read with any seriousness–amongst the editors no less. Meanwhile, there are better journals for my work that I have published in, but for some strange reason people think of CPR as setting the standard in my field. It's just an institutionalized habit as James or Dewey might say. Some journals, like CPR, function more to generate prestige of the in-group than actually gatekeep good scholarship.

    Reply
  14. Brian Leiter

    That's unfortunate you're not even getting serious referee reports. But for the sake of others, Continental Philosophy Review is not take seriously outside the "SPEP" circles, and it has never in the last thirty years published a serious article on Nietzsche, for example. So please bear that in mind!

    On SPEP: https://leiterreports.typepad.com/blog/2014/11/what-is-spep.html

    —–
    KEYWORDS:
    Primary Blog

    Reply

Designed with WordPress