The problem of incompetent peer-review

Question

In which fraction of the recent papers is significant incompetence of review(s) observed in natural sciences? In my searching, I've only come across one example of such a review... is anyone familiar with any others?

Answer 1

It's difficult to provide any sort of general assessment across all fields. My experience in peer-reviewed publication, however, has led me to conclude that truly incompetent peer reviewers are much rarer than it often "feels" to authors. It's always fun and cathartic to slag on Reviewer #3, but I disagree with your assessment of what makes an incompetent review.

Scientific communication is not a one-way street, although people often approach it in this manner. When you are writing an article, it needs to be well-matched in presentation to its audience. Comments of the form that you discuss typically reflect a disconnect between author and audience, resulting from one of two failure modes in the paper:

• The material may not be presented as well as the author thinks. We get very familiar with our own work, and often start losing track of all of the assumptions and delicate connections between different aspects of the work. It's hard to linearize the complex tangle of relationships in most work into an accessible form.
• The presentation may be good, but poorly matched to the audience. Different communities have different things they are looking for in papers and different contexts against which a manuscript will be judged. Often a paper that will be lauded by one community will be hated by another because the paper isn't appropriately presented for that community. This doesn't mean anything is wrong with the community, but rather with the authors' understanding of that community. For example, different aspects of some interdisciplinary work of mine have been published in both the programming languages and synthetic biology, and the two papers are utterly foreign to one another even though they are about essentially the same work with the same piece of software.

A truly incompetent review, on the other hand, is one that doesn't bother to particularly discuss the paper at all. For example, as a program chair a couple of years ago, I received the following review for one of the conference papers, as presented here in its entirety:

Nice work.

The reviewer was clearly happy with the paper, but this review gave essentially no meaningful input to the decision process.

So, after all of that preamble, let me return to the quantitative aspects of your questions:

• My observation has been that the number of truly incompetent reviews is vanishingly small. Editors and conference chairs generally deal swiftly with reviewers who fail to live up to minimal standards.
• Reviews showing a frustrating disconnect between reviewer and authors, like you present in your question, have been steadily decreasing over time as my skill as an author increases.
• I spend very little of my time appealing or writing rebuttals---probably less than an hour per paper, on average. I spend much more time than that revising in response to reviewer comments, however, and my papers are generally much better for the work.

Answer 2

Let's break the possibilities in two, shall we?

1. The reviewers are indeed incompetent. If it is a journal, answer the remarks. If it is a conference, submit somewhere else. It happens. However, this possibility is rather unlikely and will add nothing to you as a researcher, so..

2. Your paper is 'bad'. Don't be angry, but from your question, it seems like you failed to previously address the remarks. Most of the really good papers I've reviewed answered my potential remarks in the paper itself. For instance, they said that ""The applied method is wrong"", therefore, it would be good for you, and your paper, to have a paragraph somewhere saying "While it may appear that the method is wrong, we prove that it is indeed right because....". That way, not only you already cleared any ambiguities, you demonstrate that you reflected on the contents and possible ramifications of your work and that is important.

   You might have made some wrong assumptions about your audience, for instance, assuming that they will have a higher level than the reviewers actually have. This is a bad assumption, in general, because it is pretty rare to get reviewers/audience with a proper level. You really have to "take you audience by the hand" and guide them through the work, making the minimum amount of assumptions possible (leeway here if you don't have enough room).

Long story short, your paper let the reviewers misunderstand the contents, and that's bad.

Keep in mind that the writing itself is more important than the contents.... yes, I know, but it is true. It's easier to get accepted when you submit a very well written paper, on some not-that-good ideas, than a revolutionary idea, poorly written. Indeed, if the idea is that different (which is very good), the writing part has to be even better, to surpass the bias people have against new stuff and make them understand. (we usually try to understand stuff based on what we know, and new concepts that contradict it are harder to accept).

Of course, without properly reading the paper, this is mostly guesswork, but I've seen it enough to guess with some certainty. Nevertheless, if you paper is good, it will get better by doing this stuff, so no harm done.

---

That said, I'm not sure if I have n publications in the last y years and I don't care to count (my name is there, my scholar exists, have fun). Sorry, but you don't get to rant and pick who answers it, that's not how the world works.

I know you are probably fuming right now, because none of what I said is what you wanted to hear. Please, before you answer, calm down. I've been there, done that, it wasn't good for anyone. I'm on your side here, otherwise I could've just downvoted the question, which would be justified, from your tone alone and moved on. This is not the place for those angry rants.