10

(My field is Computer Science, where conference publications are the norm.)

I reviewed a paper submitted to conference A six months ago. The paper was borderline, and was eventually rejected. I wrote a reasonably lengthy review.

I've now been asked to review a re-submission of the same paper from the same authors for conference B. At first glance, the differences between the versions A and B are quite minor.

While I would have no problem reviewing the re-submission, I am worried about any bias (real or potential) that could occur from having access to a previous version of the work. This may be seen as preventing version B from standing on its own merits and having a "fair" shot at this conference.

It's possible that my review of paper B will have the same comments as paper A, or at the very least my review will be written in a similar style. As such, while the reviewers are anonymous, it's quite possible the authors will realize the same person reviewed the two versions of the paper.

Should I accept this review, or suggest an alternative referee?

Felix
  • 615
  • 1
  • 4
  • 11
  • Possible duplicate: https://academia.stackexchange.com/questions/7200/asked-again-to-review-a-paper-when-the-authors-dont-wish-to-modify-it – StrongBad Apr 17 '18 at 12:56

2 Answers2

17

Yes, accept. You can start your review by stating:

I reviewed version A of this paper @ [previous conference]. The differences between versions A and B are quite minor. My review remains largely unchanged:

You can then provide a revision of your previous review.

user2768
  • 40,637
  • 9
  • 93
  • 144
  • 4
    Is there a chance that this will bias acceptance at B - since you are telling them A rejected the work already? – anything Apr 17 '18 at 10:19
  • 1
    @anything Reviewers write reviews independently, so the OP's review will not bias other reviews (before the discussion phase). I like to think that there is no bias during the discussion phase either, but perhaps that's optimistic... I suppose any such bias could be avoided by providing just a revision of the previous review, but then the authors might think the OP's review is lazy, because it is largely copy-and-paste, whereas the above opening makes the OP's position clear. – user2768 Apr 17 '18 at 10:26
  • 6
    "I reviewed version A of this paper @ [previous conference]. " Careful there, this has a high risk of revealing the identity of the PC member who assigned the review. – lighthouse keeper Apr 17 '18 at 11:16
  • @lighthousekeeper No more than "providing a revision of your previous review" does. PCs usually have sufficient overlap to avoid identification, there's also an aspect of denability for PC memebers (e.g., I didn't review it, a third-party did). – user2768 Apr 17 '18 at 11:39
  • 1
    @lighthousekeeper The list of PC members isn't secret and I don't see how providing the text of a previous review reveals any information at all about which PC members might have handled the paper. – David Richerby Apr 17 '18 at 15:07
  • @DavidRicherby The intersection between PCs may allow some interference of the PC member that handled the review. – user2768 Apr 17 '18 at 15:16
  • 1
    @user2768 There might not be any intersection. And even if there is, a particular reviewer might be a natural choice: I've reviewed a few papers more than once each and I don't remember it being the same PC member each time (and they've certainly never said, "Hey, you reviewed this paper already for me for conference X; any chance you could do it again?" which you might expect if it was the same person). And, ultimately, so what? The author can often make a pretty good guess of which PC members would be interested in their paper and may have even chosen the conference on that basis. – David Richerby Apr 17 '18 at 15:20
  • @DavidRicherby Sorry, I don't understand the point you're trying to raise. Are you claiming providing the text of a previous review reveals [no] information at all about which PC members might have handled the paper even when there's an intersection? – user2768 Apr 17 '18 at 15:24
  • 1
    @user2768 "I submitted this paper to two conferences and received a review from the same person on both occasions, therefore the paper must have been handled by the same PC member" simply is not a valid deduction. Even if there is an intersection, it's completely possible that the paper was handled by different PC members each time. Indeed, more than once I've been asked to review a paper by two members of the same PC. I'm really not seeing an issue with providing the text of previous reviews. If you're worried about it, provide it as confidential material for the PC only. – David Richerby Apr 17 '18 at 15:37
  • @DavidRicherby I don't believe anyone made that deduction. But, if you received reviews from different conferences and the contents of those reviews overlap, then they are surely from the same person (I've not heard of plagiarized reviews). – user2768 Apr 17 '18 at 15:44
  • @user2768 I you receive substantially the same review, of course you can legitimately infer that it was written by the same reviewer. But the identity of that reviewer is not revealed and you were talking about making inferences about PC members, not reviewers. – David Richerby Apr 17 '18 at 16:15
  • @DavidRicherby The identity of the PC member might be inferred if the PCs intersect on a single member. Of course, the inference might be incorrect, but, in some case, there might be reasons to believe it is correct. – user2768 Apr 17 '18 at 16:19
  • 3
    @user2768 Now we're just going round in circles. Who cares about the identity of the PC member? And who cares about preventing people from making deductions when everybody should know that those deductions are false? I really don't see where you're coming from. – David Richerby Apr 17 '18 at 16:24
  • @DavidRicherby I really don't see where you're coming from: Revealing (with a decent probability) the identity of the PC member that handled a paper violates that member's anonymity. That's surely an issue? – user2768 Apr 17 '18 at 16:39
  • @lighthousekeeper fancy offering your thoughts? The discussion arose from your comment. – user2768 Apr 17 '18 at 16:39
  • @user2768 I've explained multiple times why it doesn't reveal anything at all with any reliability. – David Richerby Apr 17 '18 at 16:45
  • @DavidRicherby You're arguing that leaks aren't reliable, that's rather subjective, as is my counter-argument that leaks reveal something. In any case, I don't think either of us has evidence to definitely conclude either way. – user2768 Apr 17 '18 at 16:56
  • 1
    I agree with @user2768. If the overlap between both PCs is one member, then it's reasonable to conclude that the paper was handled by that member, even if that conclusion is not 100% reliable. This kind of conclusion making is called inductive (as opposed to deductive) reasoning, and people (including judges, scientists, engineers and physicians) do it all the time. – lighthouse keeper Apr 18 '18 at 08:14
8

Did you receive the full manuscript or just an abstract when you were invited to review? In some fields, potential reviewers are sent the abstract alone, and based on this they choose whether to review. Only if the reviewers who consent are sent the full manuscript.

If this matches your case, you should go ahead and review. If not, consider the following:

(1) Did most of your review comments in the first review deal with accuracy and use of good scientific methods?, or,

(2) Did most of your review comments in the first review deal with originality, scope and relevance?

If the answer to (1) is yes, then you should review, and mention your previous review as suggested by @user2768. The reason is simple- if there are factual mistakes, they ought to be pointed out, irrespective of history. In fact, it shows that the authors are trying to conceal mistakes/shortcomings and hoping to get lucky with a lenient review.

If the answer to (2) is yes, then maybe forget about the first review, and evaluate how different the two conferences are. It is possible that one may demand greater originality and the other may demand greater rigour, and so on. Similarly, scope and relevance expectations could be quite different. If you find yourself unable to evaluate this, possibly you are being biased by your previous review, and you should consider declining the review. Otherwise, by all means, accept the assignment.

AppliedAcademic
  • 13,178
  • 4
  • 36
  • 69
  • In both cases I received the full paper with the review request. Most of the comments were about originality (2), and since both conferences are relatively similar, I worry that my second review would be similar to the first since the two paper versions are similar. – Felix Apr 18 '18 at 07:46
  • If its an originality issue, and a borderline case, I think I would give the benefit of doubt and let it pass on to another reviewer (i.e. refuse to review). Probably depends on how strongly you feel about the (lack of) originality - if you have a strong opinion, put it forward, if ambivalent, refuse to review. – AppliedAcademic Apr 18 '18 at 09:22