27

A colleague recently submitted an article to a reputable journal. The article went to 6 reviewers, one of which completed the review and the other 5 of which completely ignored the request. The ignored requests were not declined, they were ignored such that 3 weeks passed and the requests finally timed out in the system before new review requests were sent out by the editor.

To me, it seems unethical to ignore a request rather than to decline to review. Under a decline, the article can immediately go to new reviewers. Under an ignore, it must time out.

I realize that it is possible that all 5 ignores were passive ignores, where the ignoring person never even saw the request for whatever reason. However, let's assume that the requests were actively ignored. That is, each person saw the request and chose to ignore it.

This anecdote brings up the following hypothetical questions:

  1. Putting aside the important fact that peer review is what keeps the scientific community running, are there any short term repercussions for those that ignore requests? For instance, if one of the reviewers who ignored my colleague's request were to submit an article today to the same journal and it were put on the same editor's desk, would there typically be any bias against it? Should there be?
  2. If there are repercussions, will they depend on how well established the ignoring person is in their field?
  3. What fraction of ignores are active ignores?
SDiv
  • 379
  • 3
  • 4
  • 10
    "Peer review is what keeps the scientific community running"??? – paul garrett Mar 09 '16 at 13:26
  • 1
    @paul garrett: Replace the word "running" with whatever you feel makes the sentence acceptable. Perhaps "healthy", "honest", or "credible". Considering that peer review is used in every(?) scientific field implies that it is has been deemed a necessary part of the scientific process by the majority of scientists. – SDiv Mar 09 '16 at 13:36
  • 2
    Well, I don't think anyone "deemed" it anything: it is an artifact of the once-upon-a-time literal publication possibilities, which were genuinely a bottle-neck, since they did all the type-setting, too, etc. There was no other way to literally publish. Now there is (a.k.a., "internet"). This "impact-factor" stuff is of very recent vintage, and is promoted by traditional publishers who make money from "managing" such things. – paul garrett Mar 09 '16 at 13:55
  • @paulgarrett This is a bit off-topic, but I think that peer-review still lends a certain credibility (that is in most cases justified, although reviewers are sometimes pressed for time). – Ajasja Mar 09 '16 at 14:19
  • 7
    @Ajasja, there is an obvious structural problem: reviewers are (and should be) anonymous. Why should I trust an anonymous person's opinion of a paper? Because one of the editors chose them? Which editor? Etc. "Peer-reviewed publication" is almost all about status, not credibility. Most published things are of little consequence to anyone else beyond the status-enhancement for the author ("making a living in academe"). If it matters, I certainly want to verify things for myself. Further, many more-important "peer-reviewed" papers are unreadable, riddled with (probably correctible) errors, etc. – paul garrett Mar 09 '16 at 14:24
  • 2
    Some papers are so clearly out of my scope that one wonders how much time the editor spent finding a reviewer or they must be desperate (probably for a reason). I find that quite unpleasant and I am tempted to ignore these (but I don't). Reputable or not, this whole peer-review process, was invented in the US and spilled over to Europe, has become a huge time-waster. Instead of writing my own papers, I get the job to correct some random authors' papers who do not bother - even in essentially content-wise good ones - to do a final check because - so it feels - the reviewer will find the typos. – Captain Emacs Mar 09 '16 at 14:44
  • 3
    When there is no reviewer between the author and the publication, the author feels that they are fully responsible for every detail, and the paper will appear with them being exactly responsible for every bug. The review process invites sloppiness and dilution of responsibility, but is a natural consequence of adopting a publish-or-perish mentality and of hiring committees delegating the decision of the quality of a candidate to unknown reviewers rather than taking responsibility for their own decision. I often get commended for meticulous reviews; but 80% of it should not be my job. – Captain Emacs Mar 09 '16 at 14:54
  • 2
    @CaptainEmacs Invented in the US? According to Wikipedia, the "first record of an editorial pre-publication peer-review is from 1665 by Henry Oldenburg, the founding editor of Philosophical Transactions of the Royal Society at the Royal Society of London. The first peer-reviewed publication might have been the Medical Essays and Observations published by the Royal Society of Edinburgh in 1731." – Uwe Mar 09 '16 at 17:22
  • 1
    @Uwe You are right. "Invented in the US" was me spouting nonsense. However, what I should have said, more carefully, it is my impression that peer-review became ubiquitous only through increasing US influence on the science market. The classic example is of course Einstein who experienced peer review first in Princeton in the 30s (I believe) and was quite ungracefully surprised by it. It didn't seem to become a continental habit until quite some time after the war (even in the UK). See also http://theconversation.com/hate-the-peer-review-process-einstein-did-too-27405 – Captain Emacs Mar 10 '16 at 00:13
  • 2
    I suspect reactions to the seriousness of this problem will depend on whether you are in a field with review times measured in days or review times measured in years (so the relative effect of a 2 week delay from failure to respond is different). It will also depend on whether one is in a field where people read 100 papers for every one they write or one where people read 1 paper for every one they write, so that the usefulness of the peer review filter is different. – Alexander Woo Mar 10 '16 at 00:38
  • My understanding of the verb ignore is that it can only be done actively. My dictionary even has disregard intentionally as a synonym. – silvado Mar 10 '16 at 12:21
  • 1
    @silvado, while you're not wrong there's room for a wide range of levels of intent or lack of it. Everything from "Asking for a review again, I'll delete the email without reading it, that'll show 'em", to "A review request, I'll get right on it when I've done this marking" by which time it's scrolled off-screen not to be seen again until term is over, by which time the recipient is too late to decline. – Chris H Mar 10 '16 at 16:34

4 Answers4

27

It seems this editor has a problem with their workflow of inviting reviewers, a very common problem:

  • The editor is recruiting reviewers via "opt-out" when "opt-in" would be more appropriate.

If the reviewers were asked to indicate willingness to perform a review prior to actually doing the review, then the editor could detect "ignores" of both types and find willing reviewers much more quickly. Consequences for agreeing to perform a review and then not submitting a timely report would be appropriate.

But there is nothing unethical about ignoring a review request. The potential reviewer is under no obligation to take any action at the behest of the editor -- it is fully reasonable to treat unsolicited requests as spam. And it would be unethical for the editor to take any negative action against the unresponsive potential reviewer who has never accepted the task in the first place.

Of course, the story is different for reviewers who have agreed in advance to performing a certain number of reviews. But the workflow should still be based on positive acknowledgement that the materials for review are received. The only difference is the editor's action subsequent to not hearing back -- in case the reviewer has previously committed to accepting a certain number of review tasks per year, then the editor can try a different contact method instead of assigning a different reviewer.

The bottom line is that opt-out sucks.

Ben Voigt
  • 2,525
  • 16
  • 19
  • 12
    +1 for But there is nothing unethical about ignoring a review request. Personally I always respond to such requests in a few days at most, but I will defend to the death people's right to ignore unsolicited emails. It's 100% the journal's and editor's responsibility to set up an efficient workflow that is not hindered by the lack of a timely response. – Dan Romik Mar 09 '16 at 18:55
  • 2
    I disagree. I personally think it is unethical. I use a simple, well known criterion known as the golden rule. Putting myself in the shoes of the author, I ask "Would I want my manuscript to be actively ignored?" The answer is no, and therefore conclude that actively ignoring a request is unethical. – SDiv Mar 09 '16 at 19:55
  • Further, receiving a request to review implies that one has, at one or more times in the past, submitted their own articles under the full knowledge that it will be sent for anonymous, unsolicited review. That is, by submitting your own paper for review you are entering into an implied social contract to be a reviewer yourself at some time in the future. You are "opting-in" by submitting your own work. – SDiv Mar 09 '16 at 19:56
  • 6
    @SDiv So you think that editors never contact anyone off the list of "suggested reviewers" supplied by the author? The potential reviewer may well be fulfilling their end of the social contract by performing reviews for other journals. I stand by what I said: the editor is in no position to obligate anyone to perform a review, and a potential reviewer is under no requirement to respond in any way if they haven't agreed to do it. – Ben Voigt Mar 09 '16 at 20:38
  • 8
    @SDiv it sounds like you've never been so busy and overwhelmed by emails and other duties that you had a hard time were struggling to respond to all the people who were making demands on your time and attention. The fact is many academics are in such a situation, and some handle it by ignoring unsolicited emails. The golden rule is a two way street, and the same logic you're using also implies that it's unethical to send someone you have no professional working relationship with (the majority of reviewing request) an email that is phrased in a way that creates an implied obligation to respond. – Dan Romik Mar 09 '16 at 20:41
  • 10
    @SDiv and your second comment is just nonsense. You are free to think what you wish, but I consider myself bound only by explicit commitments I've made (e.g., returning a referee report by a certain date). If someone wants to think I'm bound by an "implied social contract" I never agreed to nor recognize the existence of, that is entirely their problem, not mine. – Dan Romik Mar 09 '16 at 20:43
  • 1
    +1 for the comment "The golden rule is a two-way street". Indeed. I still try to respond to requests, being an editor myself, but sometimes I wish there was a bandwidth limitation on emails that does not probe the limits of that imposed by my eyes, brain and fingers. – Captain Emacs Mar 10 '16 at 00:17
  • 2
    And @SDiv: some of the papers I get on my table are an insult to my time. I am not talking about legitimate intricacies of the argument or high-level questions probed during the review - I am talking about shoddy work in otherwise scientifically legitimate work. Sometimes it feels that I am to do the supervision job of the PhD students of some 3rd party supervisor who does not do his job properly. – Captain Emacs Mar 10 '16 at 00:25
  • @SDiv And, since I believe these are students, I do not even dare to properly vent and force myself to be polite, helpful and constructive. In the meantime, the editor who did not dare to do a desk reject, has offloaded his responsibility to the reviewer, who, if he is like me, will not say no if he has the capacity and expertise, even if he knows what nightmare the review will be. Meanwhile, while doing the supervision job for somebody else, my own students have to wait longer for their feedback, because I try to be meticulous in my comments and spare their reviewers what I am going through. – Captain Emacs Mar 10 '16 at 00:29
  • 1
    @SDiv: in short - by claiming it is unethical, you attempt to externalise the cost of processing a paper (even if it is just deciding whether or not to take it on). I am totally with Dan Romik on that. I will do it, because young academics depend on that model, but I do not accept the "unethical" tag on those who refuse. Perhaps if more people would refuse service, the system would revert back to the model where everyone is fully accountable for their own work for better and for worse, with no one else to blame. In one year, where I accepted a huge number of papers to review, 2% felt worth it. – Captain Emacs Mar 10 '16 at 00:37
  • I don't think anyone is ethically bound to participate in the peer reviewing process - as long as they do not submit papers for peer review themselves. – Arend Bayer Jan 12 '24 at 14:36
  • @ArendBayer: See my earlier comment: The potential reviewer may well be fulfilling their end of the social contract by performing reviews for other journals – Ben Voigt Jan 12 '24 at 16:02
13

I would presume that there is not much difference between active and passive ignore, it is probably a mix of both (e.g., the reviewer received the request but forgot to followup). For this reason, there are usually very few ramifications for such a lack of followup.

However, it is quite common that review systems maintain statistics regarding each reviewer (such as number of reviewing assignments accepted/declined/unanswered, as well as average review time). In such a situation, not answering requests will make it less likely that a reviewer will be asked to review in the future. Apart from this, I do not believe that there are any negative consequences. Furthermore, reviews and review requests are usually blind, such that only the editor of a paper knows which reviewer declined to answer a request.

mdd
  • 1,236
  • 1
  • 7
  • 14
  • 1
    So, ignoring requests makes it less likely to receive requests. From a purely selfish point of view, this seems positive. If a person doesn't want to receive requests in the future, they simply ignore the ones they get now. This will have no effect on the probability of their own papers making it past the editor and on to review, and they will feel safe in the knowledge that someone else will take care of the reviewing responsibilities. This seems like a "diffusion of responsibility" issue will eventually pop up (see Wikipedia). – SDiv Mar 09 '16 at 10:49
  • 11
    @SDiv: editors do notice ignored review requests (and low-quality reviews). And editors are typically respected and well-connected people in their fields. Yes, low quality reviewers may get fewer review requests, but they will also not be asked to collaborate, be invited for keynote lectures, be asked to chair a session or to fill a vacant editorship. Prospective students will be counseled against working under them. Most of this not because of ignored review requests, but because ignored requests are a symptom of low dependability. – Stephan Kolassa Mar 09 '16 at 11:14
  • @Stephan Kolassa: Indeed. This leads into question number 2. Those who are already well-connected and established can feel free to ignore review requests. – SDiv Mar 09 '16 at 11:26
  • I would classify example given by @mdiener ("the reviewer received the request but forgot to followup") as an active ignore. The reviewer placed so little thought and effort on the review that it was forgotten. Further, I believe it is typical to receive reminder emails, in which case active ignore is again implied unless the reviewer does not read/receive their emails at all. – SDiv Mar 09 '16 at 12:38
  • 2
    @SDiv Your definition of "active ignore" seems to imply a bit of malice on the part of the reviewer. I am not sure if this distinction between active and passive is very useful, since there are many reasons a request could be ignored without malice (spam filter, software error, wrong email address, ...), and the reason for an ignore is impossible to determine even for the editor that sent the request. However, if there is a pattern of many ignored requests, I would assume that the system or editor will remove the "offender" from the list of potential reviewers. – mdd Mar 09 '16 at 13:24
  • @mdiener: If you see a request and do not respond to it then you have actively ignored it. This action is, in my opinion, unethical. Perhaps you also feel this is malicious, I don't know. If you do not see the request then you have passively ignored it. This is not, in my opinion, unethical. I agree that the editor doesn't know what type of ignore it was. The point is that unethical behavior in this instance may have no negative repercussions and may actually be rewarded, depending on your point of view. – SDiv Mar 09 '16 at 13:45
  • @SDiv: I would change the language, as in "ignore": seen but not responded (for whatever reason), and "not seen/received" (for whatever reason). I think an ignored review request is not necessarily an unethical (or even malicious) behavior, in the same way that a lack of reply to a question received via email is not unethical IMHO. – mdd Mar 09 '16 at 14:03
  • 3
    I actively ignore requests from lower ranked journals, and will always respond to requests from places where I have published in. – Prof. Santa Claus Mar 09 '16 at 20:25
5

Actually, there can be some detrimental effects for faculty who actively refuse to participate in peer review processes. For instance, the documentation for promotion and tenure at some universities requires you to list your reviewing activities. If you don't have any, that means your documentation will have an unexpected blank space in the "service" activities. This isn't normally enough to deny someone promotion or tenure, but it is enough to warrant comment from the typical review committee. ("The rest of us are doing this—why aren't you?")

aeismail
  • 173,481
  • 34
  • 418
  • 736
  • I imagine that someone could have a full list in that category, even if they didn't perform reviews for every editor that contracted them. – Ben Voigt Mar 09 '16 at 20:40
  • @BenVoigt there is certainly no obligation to perform reviews for every editor that asks, however I think it is mean-spirited if not unethical to simply delete a review request without hitting the "decline" link first. It is not the editor who suffers from having to wait for invitations to time-out, but the author. In my experience as an editor, this is the largest factor contributing to slow reviewing processes. It is all very well saying that editors should only send review requests to people who have opted in, but in practise this would result in the same few people being asked constantly. – Significance Mar 10 '16 at 00:26
  • 1
    @Significance I didn't mean to suggest that contacting potential reviewers needs to stop, but that the request should come with an "consent to review" link not a decline link, and that the editor cannot consider reviewers found unless they have consented. – Ben Voigt Mar 10 '16 at 01:32
  • @BenVoigt they usually come with both a "consent to review" link and a "decline" link. Editors do not consider reviewers found until they have consented, but they still need to wait for reviewers to respond one way or the other, or else just wait until it is clear that the potential reviewer is not going to respond. – Significance Mar 10 '16 at 02:15
  • @Significance: The question says that the editor assumed the first group was writing reviews, and didn't contact more potential reviewers until the reviews were overdue. That's the problem... – Ben Voigt Mar 10 '16 at 02:27
  • @BenVoigt I can't see where it says that in the question. It says 6 potential reviewers were initially contacted (the editor would have been hoping that two of those would accept) and the editor waited 3 weeks before contacting more potential reviewers since 5 of the 6 never responded. This is reasonable, as reviewers do not always respond immediately and it is awkward if you end up with 5 acceptances and have to withdraw requests after reviewers have already agreed and perhaps stated work. – Significance Mar 10 '16 at 06:34
  • 1
    @Significance: If reviewers start doing the review without hitting the "I commit to this review" button first, that's their own fault. Any reviewer who hasn't said "yes" within 1 week isn't doing it for whatever reason -- no time, no interest, didn't see it because of spam. It doesn't matter. (In addition, the journal should know when reviewers use the link to download the complete paper -- the email should contain only enough information to accept or not, which should be possible from the title + abstract, no?) – Ben Voigt Mar 10 '16 at 06:50
  • @BenVoigt so you'd argue for a one week time out instead of a three week time out. That is probably a good idea, though I admit I have sometimes agreed to do a review only after a reminder at the two or three week mark myself, when busy . – Significance Mar 10 '16 at 10:40
0

For instance, if one of the reviewers who ignored my colleague's request were to submit an article today to the same journal and it were put on the same editor's desk, would there typically be any bias against it? Should there be?

The question asks about what should be done, so I'll argue from an ethical perspective what I think should be done, and how I try to handle things myself. The result is that yes, there should be bias, and it stems from the idea to give people back what they gave you.

As an editor, you can distinguish reviewers based on two dimensions: How timely they respond to requests and complete their reviews, and how much detail and constructive comments their review contains. Let's consider the situation when an author submits a paper to be sent for peer review, and the author has acted as peer reviewer before. If the author has a positive record as a peer reviewer, I think the editor will have the moral obligation to also find peer reviewers with similar qualities. So if the author usually responds timely to review requests and finishes reviews in time, the editor should invite reviewers who are expected to handle things timely. If the author usually gives detailed, constructive comments on a manuscript, the editor should make an effort to also get reviews with similarly detailed and/or constructive comments.

Note that this does not take any bias towards the scientific quality of the manuscript, and of course should not affect the decision for acceptance or rejection.

silvado
  • 18,750
  • 1
  • 67
  • 87
  • -1. I disagree with your advice about how to treat authors who were reviewers, but regardless, it doesn't even answer the OP's actual question, which was about consequences for people who were asked to review a paper but never even responded to the request. As pointed out by others this could be the result of never actually receiving the email. – Dan Romik Mar 10 '16 at 17:15