16

I wrote a thorough review that included many suggestions for improving the methodology, and criticisms of their analysis. Initially, the main problem with the paper was that the authors only reported significance tests (something like p-values) without reporting effect sizes. Since they provided their data, I calculated the effect sizes and found that they were tiny, and "significance" was only achieved through some dodgy methodological choices (artificially inflating the sample size). The second reviewer brought up some of the same concerns and the paper was rejected.

I was concerned about writing such a negative review, because I know it can hurt. But I did my best to write in a positive tone, and to make as many suggestions as criticisms.

I now see the paper has been posted as a pre-print, without any changes. If I were to see it posted in a journal, I would honestly say that it should be retracted because the conclusions are not supported by the data.

What do I do in this situation? I consider my options to be:

  1. Contact the authors and ask for a discussion. Perhaps some of my criticisms were wrong, and the authors did not have a chance to write a rebuttal since the paper was rejected.
  2. Wait for the paper to be published, and contact the editor.
  3. Post my review to the pre-print as a response.
  4. Forget about it, because life's short and who cares.

Update: Thanks for the enlightening discussion. I'll stick with 4 for now, since I don't know what the fate of the article will be (will it get published? will it be updated?). While nobody seems to suggest 1, I have a morbid curiosity about what kind of response I would get. This could always be followed up by 2 or 3. If I'm luck I'll have better things to do.

Noil
  • 277
  • 2
  • 7
jomimc
  • 349
  • 1
  • 1
  • 9
  • https://academia.stackexchange.com/questions/135326/journal-published-a-paper-ignoring-my-objections-as-a-referee Related, might be duplicate. – Allure Jan 30 '24 at 03:49
  • 2
    Thanks, this is related. However there are enough differences between the questions that I would not say it's a duplicate. In my case the paper was rejected by the editor based on the reviews, and has now been posted as a preprint. Also, I'm considering a wider range of options as a response, including communicating with the authors. – jomimc Jan 30 '24 at 04:11
  • 2
    It is well possible that the next review process will request revisions from the authors before publication, and that the paper will be substantially improved when finally published in a journal. I'm just writing this because your question makes me wonder whether you are aware of that possibility. The community is normally aware that there is no quality control for preprints, so such issues in a preprint that is not otherwise published will arguably not do that much harm. – Christian Hennig Jan 30 '24 at 16:59
  • 1
    I'm not sure what's the norm in your field, but in my field anyone is free to post anything as a pre-print, in general, right? Even this includes unreviewed work. Maybe they wanted to post that pre-print before, but was held because it's still under review and the journal/conference doesn't allow posting a pre-print, so after it's done, they posted it as a pre-print, with the intent to update it with a revision later, etc. This is why general advice is to not cite pre-print, because it might be unreviewed, or incorrect, etc. – justhalf Jan 31 '24 at 08:25
  • 9
    isn't a pre-print a pre-print - a version of the paper that is not final? as I understand it, peer review is not required to put a paper on a pre-print site, and some of them even carry crackpot theories. – user253751 Jan 31 '24 at 12:29
  • At this point I think pre-print servers are just misleadingly named. I know many colleagues who consider the pre-print version final, because it can be updated even after publication. On the other hand, one can upload pretty much anything there. So if someone wants to upload something questionable to a pre-print server, they'll be in good company out there on the internet. No reason to lose much sleep over. – Scriddie Jan 31 '24 at 14:16
  • 1
    A referee only makes suggestions for changes, the authors may or may not act upon these suggestions. – Tom Feb 01 '24 at 20:36
  • @user253751: A pre-print is not final w.r.t. the format, the exact scope, the phrasing, etc. It's final w.r.t. the claims having to be supported by the data. – einpoklum Feb 01 '24 at 23:29
  • @einpoklum but it's also prior to peer review, which means you can just pre-print anything if you have enough reputation to publish on the pre-print site, which is very low on some sites. – user253751 Feb 05 '24 at 23:48

5 Answers5

47

You have fulfilled your obligation as a reviewer ... but the authors whose work you reviewed are under no obligation either to agree with you, or to take any notice of your opinion. Indeed, it's always possible that, unknown to you, they received a very positive review from someone else!

Sure, if the topic interests you, and you want to post some kind of response, go ahead, but remember ... there are probably many more interesting things for you to spend your time on, including many more interesting things with which you have some disagreement and where your engagement will enrich your life. (As you remarked, life's short)

CrimsonDark
  • 11,150
  • 1
  • 28
  • 64
  • I agree that the authors do not have to agree with me, and I am probably wrong on at least some points, however the other reviewer raised similar concerns and the paper was rejected. I can't imagine a competent reviewer failing to raise similar points (e.g. not reporting effect size). I genuinely think the work has such severe flaws that it should not be published anywhere in its current form. – jomimc Feb 05 '24 at 03:43
  • That said, I do have more interesting things to work on... – jomimc Feb 05 '24 at 03:44
15

Think about how much of your own time you are willing to put into this. Assuming your conclusion about the effect sizes was correct, and the paper was methodologically dubious, I don't see what #1 would accomplish, since they have already gone ahead and published the preprint despite your reservations.

I have noticed that often people put up preprints (yes, even for somewhat incomplete, if not entirely dubious studies) to unofficially, but publicly, lay claim to a certain research problem — so they can later say, ours is the first result demonstrating effect X.

I can't claim to know their intention, but I would suggest skipping #3, give them the opportunity to revise and resubmit. Perhaps if you still feel strongly about it (if and) when it is published, you could go ahead with #2. Otherwise, there is always #4.

Just my two cents.

Ivory Tower
  • 730
  • 4
  • 12
  • 1
    Contact the editor to tell them what, exactly?? If the reference it to the journal that turned them down, they rejected the paper, and have no dog in this fight, at this point. If the reference is to the editor of the published paper, it's gone through their review process. What action would the new editor take? There's really no point to 2. – Scott Seidman Jan 30 '24 at 19:22
  • Scott, suppose that the authors really have no intention of making any changes to their work -- for whatever reason. The authors could try to publish someplace with a shoddy review process or the next reviewer entirely overlooks the issue OP found. In this event, while I am not specifically encouraging OP to take up arms and write to the editor of the journal with their concerns, I do think it would be nice if sometimes people did that. Notice that I say "Perhaps if you still feel strongly about it [...]". What the editor decides to do about this is a different matter. – Ivory Tower Jan 31 '24 at 00:15
  • Again, this assumes that (a) OP has the time and capacity to do any of this; (b) they are confident about their analysis concerning the effect sizes; and (c) the authors of the study did not make revisions to their work. – Ivory Tower Jan 31 '24 at 00:15
8

1 and 3 are not options since they violate reviewer anonymity. You could ask the editor you reviewed for to contact them on their behalf, but I doubt they will want to get involved.

I would not worry too much for the moment. Consider the alternative scenario that they posted their preprint before receiving your review, and have not yet updated it. (Indeed, editorial workflows being what they are, can you be sure that isn't what happened?) Clearly there would be no reason to be concerned in that case. Is the actual sequence of events so different? After all, the effect so far is the same.

It is quite possible that they haven't had time to deal with the comments yet, but needed a preprint to be available immediately for whatever reason - claiming priority, one of the authors is applying for a job, etc.

Especially Lime
  • 9,290
  • 1
  • 22
  • 38
  • 14
    The reviewer is free to forfeit their anonymity. Or they can just contact them without mentioning the review (sure they will have their suspicions given the similar remarks). – Kvothe Jan 30 '24 at 19:10
  • 4
    @Kvothe no they are not. E.g. Elsevier's reviewer guidelines explicitly say "Please do not contact the author directly." Nothing special about that publisher, just the first one I looked at. – Especially Lime Jan 30 '24 at 20:45
  • 4
    @EspeciallyLime: It doesn't seem clear at all in which sense those "reviewer guidelines" can be considered binding (or what "binding" even means here). It seems more reasonable to argue with the culture of individual fields rather than with a publisher's generic reviewer guideline (which is somehow supposed to "fit" all fields and which most reviewer probably never read). From experience I can for instance say that many mathematicians disagree with your claim that must keep anonymity after the peer review process. – Jochen Glueck Jan 31 '24 at 00:43
  • A clarification: the preprint was posted several months after the review. My guess is that they have either given up on getting it published, or else it is about to be published in a journal. Time will tell. – jomimc Jan 31 '24 at 07:09
  • Also, I would think the reviewer guidelines are not necessarily binding after a paper has been rejected. It's not an NDA, right? And if not, I don't see Elsevier coming after me for this. – jomimc Jan 31 '24 at 07:10
  • @JochenGlueck I don't think "mathematicians" is the right population here. Would math journal editors think it is acceptable for a reviewer to break anonymity? I suspect that the people who think it's ok to reveal themselves never bother to ask permission (and wouldn't get it if they did). – Especially Lime Jan 31 '24 at 08:25
  • 2
    @EspeciallyLime: Editors of math journals are not different from other mathematicians; they share the same academic socialization. I heard an editor-in-chief of a well-respected math journal say the sentence "The referee is of course not obliged to stay anonymous". On another occassion the handling editor of a paper that I had reviewed asked me (not the other way round) for permission to reveal my identity to the authors, since the authors wanted to acknowledge me for the solution of a problem in their paper that I had included in the report. – Jochen Glueck Jan 31 '24 at 09:03
  • @EspeciallyLime: Apart from that, as I indicated in my previous comment, it's not clear what "permission" does actually mean in this context and on which grounds the reviewer would be obliged to ask for permission. Is it a legal category? Or ethical? Social? – Jochen Glueck Jan 31 '24 at 09:31
  • @EspeciallyLime, good to know. Another reason never to review for an Elsevier Journal. I would never knowingly agree to such strict terms (in my opinion one would have to be crazy to agree to such a strong restriction on my free speech for basically nothing in return). To make it clear this is not at all uniform here is an example of a journal that not only enables but actively encourages referees to sign their report: https://scipost.org/FAQ. – Kvothe Jan 31 '24 at 17:00
7

There are sites for public reviews, like https://pubpeer.com. You can post your reviews there.

jerlich
  • 3,015
  • 11
  • 25
3

Post a comment. Posting of comments is explicitly supported on some pre-print servers, for instance on bioRxiv.

I think you are free to comment on the public pre-print just as anyone else would be. If you'd like to maintain plausible deniability that you were the reviewer, you could rewrite your review in different language and maybe focus only on one fatal flaw. The authors may still suspect you were the reviewer, but they would not know for sure.

You could also write a formal comment on the pre-print as a paper of your own and submit it to the pre-print server. These have been common in traditional journals for many years. For example, this ACS journal allows for "Comment" manuscripts that are "technical contributions that provide new information or an alternative perspective regarding a prior publication in the journal". I see no reason why this cannot be done for pre-prints as well, although it might require a significant investment of time on your part.

WaterMolecule
  • 3,024
  • 2
  • 13
  • 22
  • I've done this with other papers, but not possible for this one unfortunately. Otherwise good suggestion. – jomimc Jan 31 '24 at 12:11