18

I received a rejection letter from a relatively high-impact journal based on the feedback from three reviewers. After I reviewed their comments, I feared that their comments were biased and am not sure if I should appeal. The comments from all three reviewers are all negative, but one is more bias than the other.

The first reviewer said that I need to carry out a lot more experiments that will involve a lot of cost, which is not practical. However, in the comment, they only suggested one additional experiment that needed to be done. The second reviewer complained about the quality of the figures, but I think they are fixable. The third reviewer complained about the novelty of the paper but I think they did not carefully read and understand the paper.

Is it still possible to write an appeal if all three reviewers were negative about the paper since their reason to support rejection wasn't strong?

Sursula
  • 20,540
  • 8
  • 62
  • 121
user10694
  • 1,639
  • 1
  • 15
  • 21
  • 43
    I'm afraid appealing the decision will most likely be a waste of time. Some, most, or all of the feedback you obtained this time around is probably useful, so use your energy on strengthening the paper based on the feedback you obtained. Also, the editor and/or associate editor at the journal you submitted to may not truly appreciate your contributions, so do some soul searching about where a more appropriate venue will be for your paper. – Mad Jack Apr 23 '14 at 22:28
  • What if I encounter the same sets of reviewers in the other journal? If the Reviewers don't like it for various reason, they can always find hole in the paper. How can I improve the chance of being accepted? – user10694 Apr 23 '14 at 22:34
  • 34
    @user10694 Fixing the problems the reviewers pointed out will improve your chances of acceptance. That's the usual response to a negative review. – ff524 Apr 23 '14 at 22:47
  • 1
    I find it interesting that people underplay the realities of bias and prejudice in paper reviews. We all know it is unfortunately widespread in employment practices including hiring, firing and promotion. However we pretend it is unheard of in paper reviewing. How many court cases are there for unfair employment practices based on racial,gender, sexual orientation bias every year? Much of the advice here is really no better than "Take it on the chin. There is nothing you can do about it." – Simd Apr 24 '14 at 08:07
  • 3
    @felix ... based on racial, gender, sexual orientation bias. The problem with your logic, is that the reviewers do not know the OP personally. So, why prejudice against him? Isn't logical to assume that the paper was rejected because it was not good enough, instead of thinking that they rejected it because the OP was poor / ugly / unknown etc... – Alexandros Apr 24 '14 at 08:27
  • @Alexandros We have no idea if the reviewers know the race, gender or sexual orientation (or other relevant attributes) of the author do we? I find it perfectly plausible that they would know or be able to guess at least some of these however. For a randomly chosen example it is a reasonable guess that the reason was innocent. However there will be many cases where it isn't and the system should be able to address those. – Simd Apr 24 '14 at 08:31
  • 26
    @felix I think what the answers and comments are responding to is that the OP says he/she got a biased review, then goes on to describe some very standard run-of-the-mill negative reviews. The community response would (I think) be very different if the user had given a reasonable explanation as to why he/she thinks these specific reviews indicate some kind of bias. – ff524 Apr 24 '14 at 08:42
  • 20
    This may not be what you want to hear, but maybe you are biased and not the reviewers. The fact they all seem to agree on rejection can be an indication to that, though the fact that all reviewers seem to have issues with different aspects of the manuscript makes it less likely. – Marc Claesen Apr 24 '14 at 08:51
  • 1
    @ff524 That is a fair point. I think I was answering a more general question than the one the OP posed. However the general concern still remains that there is no established way to challenge bias in reviews or reviewers for many conferences (and to a lesser extent journals too). – Simd Apr 24 '14 at 09:00
  • 1
    @felix: The best way to remove bias would be to double-blind reviews, where neither the referee nor the authors know "who's on the other side." It would make it hard to practice bias based on who the author is—you only have the material in the paper to work with. That makes outright bias easier to spot. – aeismail Apr 24 '14 at 12:33
  • @aeismail Yes. I believe experiments have been done to show the difference you get in results for accepted papers between blind and non-blind reviews although I don't have the reference to hand. – Simd Apr 24 '14 at 12:40
  • 19
    @user10694 Biased against what? (Your gender, ethnicity, country, academic position, topic of research, particular approach to topic, ...?) Otherwise word biased makes little sense. Bias (see Wikipedia) is not a synonym of negative. So please, either expand or remove word bias from this question. – Piotr Migdal Apr 24 '14 at 14:47

6 Answers6

81

@Fomite's answer is spot on. I would like to expand further on the differences between a biased review, a lazy review, a significantly erroneous review, and a negative review, and which do/don't warrant an appeal.

The examples given in this post come from a contest held by the always-excellent FemaleScienceProfessor. Quite a few are real reviews, or based on real reviews.

Biased Review

Does warrant an appeal, if it's the reason your paper was rejected. Often editors are smart enough to identify a biased review on their own, and will simply discount its advice.

Example #1 (here, the reviewer clearly lets his personal goals get in the way of a fair review):

I am afraid I cannot support the publication of this paper. The authors present an idea that I myself have had for some time and just haven’t gotten around to writing it up yet. I think that when I write my paper, it will be better than this one. These authors would be well advised to wait until my paper is published, and then they can cite it.

Example #2:

Since the author is a woman, I had lowered my expectations accordingly, but the author did not even meet those. Even though I have never done experiments in my life and have never used any of these techniques, the experimental results presented were completely misinterpreted in my distinguished opinion.

Lazy Review

Does not warrant an appeal. If all of your reviewers give you a lazy review, you need to do a better job exciting the reader (thereby motivating the reviewer to carefully read the whole paper in detail because it's such compelling reading.) (Very rarely, this can be an indication of a bad journal/conference and you should find a better place to submit your work).

Example #1:

The topic of this paper is not of critical importance to furthering knowledge.

Sent from my iPhone

Example #2:

Paper Title: Linear and Nonlinear Methods to solve XXX

Review: "Why is the approach limited to linear methods, and the author does not propose nonlinear methods?"

Significantly Erroneous Review

(per JeffE's suggestion)

This is a distinct case from the Lazy Review. It does not include the case where the reviewer "erroneously" concludes that your work is not interesting or useful, just because you disagree with this conclusion. It does not include the case where the reviewer says "I am concerned about X" and you believe you've adequately addressed X.

It does include cases where a review is affected by a significant factual, logical, or mathematical error. This is grounds for appeal.

Example (the reviewer is mistaken as to the meaning of "graph" here):

Paper title: Graph analysis of System ABC shows differences in A-B connections during Condition X vs. Y

Review: This paper uses a novel method to study ABC system dynamics. However, I don't know anything about graph analysis so I'm going to interpret the paper through the lens of a different analysis method I do know something about. Thus the only constructive feedback I can give is that this paper is a poorly written explanation of a structural equation model of XYZ - the conclusions drawn make little sense given the data, i.e. they frequently refer to "graphs" but all I see are these pictures of circles and arrows.

(A minor error is highly unlikely to be the reason for a rejection. Therefore, for a minor error, an appeal is not likely to be successful - an appeal will be useful only if the point of appeal was the reason for the rejection. In the case of a minor error, I would advise to use the reviewer's mistake as an opportunity: to clarify in the text points that might lead people to mistakenly discount your results, or even to decide to ignore the criticism because it's not a common enough misconception to be worth addressing in the paper.)

Negative Review

A review that does not fit into the above categories, but still criticizes your paper, is just a negative review. It is not grounds for an appeal. It should be taken as constructive criticism, and you can use it to make your paper better.

Community
  • 1
ff524
  • 108,934
  • 49
  • 421
  • 474
  • @Cohen I can't take credit for the hilarity - the examples come from the FemaleScienceProfessor contest mentioned at the beginning. I will happily take credit for recognizing the hilarity and sharing it with you, though :) – ff524 Apr 24 '14 at 05:18
  • Yeah, I realized that just a few seconds ago, but you do deserve credit for recognizing and sharing it. Thanks for making sure I knew the source! – Cohen_the_Librarian Apr 24 '14 at 05:21
  • 3
    There are also uninformed/incorrect reviews, where the reviewer is simply mistaken. – JeffE Apr 24 '14 at 07:02
  • @JeffE Certainly. I would include those in negative reviews - no one said "take as constructive criticism" means "apply the changes the reviewers suggest." it can also mean "clarify in the text points that might lead people to mistakenly discount your results" or even "decide to ignore the criticism because it's not a common enough misconception to be worth addressing in the paper." – ff524 Apr 24 '14 at 07:07
  • But uninformed reviews are reasonable grounds for appeal, unlike informed but negative reviews. – JeffE Apr 24 '14 at 07:10
  • @JeffE Thanks, on further research I agree and have edited the post to include the "Significantly Erroneous Review" category – ff524 Apr 24 '14 at 07:45
  • I had example #1 in your "Biased Review" section when submitting to a prestigious conference and the paper was rejected. I then submitted the same paper elsewhere and the same reviewer reviewed it again. There needs to be some sort of appeal system for conferences as well as journals. – Simd Apr 24 '14 at 08:11
  • 2
    @felix I've submitted to a conference with an author response phase, which can be a nice thing. Suggest it to your friendly local conference sponsor. Though I don't know how one might "respond" to Biased Reviewer #1 :) – ff524 Apr 24 '14 at 08:15
  • 1
    @ff524 That does seem like a nice innovation. I hope it becomes more widespread! – Simd Apr 24 '14 at 08:20
  • First, I think the biased review can be easily discerned but not always explicit. The reviewers are smart to achieve their hidden motive by finding holes in your paper or purposely distort what you stated in the manuscript. For lazy review, why would the reviewers volunteer for the reviewing topics/abstract that they are not interested in the first place? – user10694 Apr 24 '14 at 18:20
  • 6
    @user10694 The reviewers are smart to achieve their hidden motive by finding holes in your paper or purposely distort what you stated in the manuscript - do you have any reason to believe this is their motive? If it's so well-hidden, how do you know and how do you propose to convince the editor in an appeal? – ff524 Apr 24 '14 at 18:26
  • 7
    @user10694 For lazy review, why would the reviewers volunteer for the reviewing topics/abstract that they are not interested in the first place (1) they were interested on reading the title/abstract but then you failed to hold their interest with the actual paper (2) pressure from editor (3) it's a student and advisor said "review this paper for me" (4) etc. (many possible reasons) – ff524 Apr 24 '14 at 18:28
  • Hilarious examples! Bet you have seen/written a couple of reviews. – Zane May 15 '14 at 07:57
76

You seem to be conflating "Bias" and "Negativity". Bias would imply that your paper was being reviewed unfairly, either because of its content, or its authors - if they were known or divined by the reviewer.

It's entirely possible, and fairly standard, to get negative reviews you disagree with. When it comes down to it, your paper was rejected, and the only time I've ever seen appeals to the editor work is for things that are flagrantly egregious and wrong, rather than just "This is weak and I disagree with it". And even then, it's usually a long shot "What's the worst that could happen, they say no again?" play.

I suspect your time would be much better spent addressing the comments of the reviewers within the paper and revising it, rather than appealing to the editors.

Fomite
  • 51,973
  • 5
  • 115
  • 229
  • 14
    +1, good response. Especially insightful: "You seem to be conflating "Bias" and "Negativity"" – Cohen_the_Librarian Apr 23 '14 at 23:52
  • What constitutes bias? If the reviewer did not read the paper and gave negative comments, that is bias and that is exactly what is happening to me. – user10694 Apr 24 '14 at 00:09
  • 17
    @user10694 That's not bias - again, that's negativity, and you don't know the reviewer didn't read the paper. Bias would be something like the reviewer knocking a paper because it didn't come from their lab, or a friends lab, or because "all theory papers are terrible". Not just "I didn't like it". – Fomite Apr 24 '14 at 00:29
  • If the reviewer read the paper, why would one of the reviewer get confused while two others don't? – user10694 Apr 24 '14 at 00:41
  • 31
    I'm sure I've read similar advice on this site, but what I've come to believe is that if someone didn't understand your writing, it's because you didn't make it clear enough. The onus is the writer, not the reader. If you can determine what it is the reviewer was confused about, write it more clearly and try again elsewhere. – Mike A. Apr 24 '14 at 00:46
  • 2
    What I'm saying is that the reviewer simply ignore the statement in the manuscript. For example, if I mentioned "something" in the manscript, in the feedback he said he did not see that "something" I have written. – user10694 Apr 24 '14 at 00:50
  • 21
    Then maybe you need to make that "something" easier to find. A poorly organized paper is a valid reason for rejection. I concur with the others: forget about repealing, commence your revising. – J.R. Apr 24 '14 at 01:29
  • 6
    A big +1 for the first paragraph. I had the identical reaction. – Pete L. Clark Apr 24 '14 at 01:35
  • 5
    @user10694 if a reviewer 'didn't understand' your paper then under most circumstances it is more productive to assume that the fault is with you: you wrote the paper poorly, so that an expert could not understand it, revise it and emphasize the parts they missed. – Artem Kaznatcheev Apr 24 '14 at 04:42
  • 1
    I generally agree, though (without changing the conclusion referring to what to do in the OP's situation) I'd like to remark that reviewers reading the paper sloppily/incompletely and then stating the lack of a definition for X (even though there is an explicit definition in the paper) as a negative point does happen. It is easy to claim the paper was not properly structured, but while in the age of full-text search that is already a poor excuse for not finding a literal definition, it usually shows throughout the whole review in such cases that the respective reviewer did a rather hasty job. – O. R. Mapper Apr 25 '14 at 22:20
20

The second reviewer complains about the quality of the figures, but I think they are fixable.

You fix the figures and then re-submit. Not a problem.

The third reviewer complains about the novelty of the paper but I think s/he did not carefully read and understand the paper.

You write to explain why you think your paper is novel. You might still have a chance.

Now,

The first reviewer say that I need to carry out a lot more experiments that will involve a lot of cost, which is not practical. However, in the comment, s/he only suggests one additional experiment that need to be done.

In my opinion, this is a strong reason for rejection. I think the reviewer has doubts about the results you have in the paper. You truly need to do more experiments to convince the reviewer and other readers the results in your paper are correct.

My suggestion is to do more experiments before you consider the appeal. Not only you want to convince the reviewer but also yourself that your paper is good. You don't want to publish a bad paper, do you?

Nobody
  • 13,796
  • 29
  • 57
  • 134
  • well, I think the first reviewer just blow the negativity out of proportion. If there were so many experiments that need to be done, why would he/she only suggested only one experiment. Also, why would he/she concern about the costs of the experiment? – user10694 Apr 24 '14 at 18:25
  • 1
    Do this one suggested experiment as proposed, maybe this will be enough if you also fix all other issues. – algorithmic_fungus Apr 25 '14 at 08:47
  • 6
    @user10694: Also worth noting: you don't necessarily have to do even the one suggested experiment, if you can come up with a different experiment that better demonstrates the same point. (This happened to me with my first paper: a reviewer suspected that the stability of the steady state I claimed was merely due to insufficient simulation time, and asked me to run the simulations for a longer time. The editor, however, instead suggested simply showing that the steady state was attracting, which would imply stability; I did that, and the paper was accepted.) – Ilmari Karonen Apr 25 '14 at 11:33
  • 1
    About point 1: In my experience, there is usually an alternative to further experiments: give an honest discussion of the limitations of the study. As a reviewer I'm usually perfectly OK with a not-so-great study as long as the authors are clearly aware of the limitations in terms of conclusions that can be drawn - in my field, noone can afford all the experiments one would like to do and we're all aware of that. But papers that, say, claim to rescue the world after having compared a single patient with one healthy volountary member of the author list raise very red flags. – cbeleites unhappy with SX Apr 25 '14 at 16:33
  • 2
    In my field, you don't resubmit to the same journal after a rejection, even if you've fixed figures and such. – PatrickT Apr 25 '14 at 19:48
17

Short version. Do not do it.

Long version: You have to understand there is something wrong with the paper. One negative review might be unlucky / unjust. But three negative reviews? That means that the paper needs some serious fixing before re-submitting again.

The other possible problem with your paper might be that you are aiming too high. Maybe your paper is actually good but not good enough for a high impact journal. That would also explain why the third reviewer just "brushed off" your paper without really reading it. Your paper simply might not be interesting enough for such a journal. This is a phase that many researchers have in the beginning of their research. Living in our research "bubble", we believe our research and results are amazing, when simply they are not. It takes some research maturity to actually understand the real "size" of our results.

Also based on your previous questions on this forum and your problems with your advisor it is obvious that you lack guidance. You are simply not objective enough to judge the merits of your research and you need someone "wiser" to judge your work hard but in a fair way. But please humble down. Do not start a war with the world. Your advisor was "bad" and you had to fight back. Now the reviewers are bad? This is really not happening. Just improve your work, your presentation of the work and if your work is good enough it will be published accordingly. If it is not published, it is nobody's fault but yours. You should at least start considering the possibility that you might actually be wrong and the reviewers are right. After all, they are more experienced than you.

Alexandros
  • 18,921
  • 3
  • 49
  • 84
8

I suspect that the reviewers are responding to an unconvincing paper. A paper that does not convince them. While you have not reported evidence of their bias (or explained the reason that you suspect that they are biased), I find myself somewhat biased by the quality of the English used in your posts. If the language in your paper, matches that in the posts, then it is unlikely to convey a convincing presentation of your research - however good the research might be. If you think this is a contributing factor to the rejection (or is responsible for the bias), then the best approach may be to enlist the assitance of a Technical Author or Technical Translator, to help you turn your paper into a convincing one for an English Language publication. Alternatively you might find it easier to publish your research in a different langauge journal. Please understand that I'm observing, not criticising. Your English is much better than my own efforts in any language other than English.

Nick
  • 81
  • 1
3

One thing I haven't seen commented on in the other answers or comments is that if you appeal claiming that their review was biased, you're poisoning the well -- you'll have accused three known and (presumably) respected reviewers of unprofessional conduct. If they didn't dislike you before this, they'll certainly have cause now.

From what you're saying, this seems to be your first submission and you yourself admit that at least some of the objections they have to your paper are fixable. Try fixing those and resubmitting. Don't start throwing around accusations of bias until you have a clear pattern, or you'll quickly develop an unwelcome reputation of your own.

Shadur-don't-feed-the-AI
  • 137
  • 4
  • 5
    You don't need to have a clear pattern of bias—you just need clear evidence. Moreover, an appeal on bias grounds would in principle be anonymous—the editor would ask for an additional review. – aeismail Apr 26 '14 at 10:48