52

Every so often, I stumble upon a question concerning "double blind peer review", i.e., the authors don't know the reviewers (as usual) but also the reviewers don't know the identity of the authors.

It's always struck me as a rather odd system, with no discernible benefits (because I am under the impression that it's often rather easy to determine who the authors of a paper are anyway), and the creation of dozens of problems with people always being worried of somehow "breaking" the double blindness. (It is not helped by the fact that this would be impossible to implement in my field, math, where everyone posts their preprints on arXiv and people don't hesitate to give talks about unpublished papers).

Is there any actual, scientific, serious research into evaluating the benefits of this system? Or is it all just inertia/tradition/good intentions? I'm not talking about armchair justifications for it with no data to support it.

  • 6
    Roughly analogous question about single-blind review: Open versus Blind reviewing process – O. R. Mapper Mar 09 '18 at 12:45
  • 3
    @Mehrdad I don't frequent websites for conspiracy theorists. –  Mar 10 '18 at 08:10
  • 3
    "a rather odd system, with no discernible benefits" I would suggest you haven't given this minimal thought if you really believe that. – neuronet Mar 10 '18 at 14:37
  • 5
    @NajibIdrissi And yet, you come here with a conspiracy theory of your own (that double blind peer reviews are not beneficial). – Agent_L Mar 10 '18 at 16:59
  • 7
    @Agent_L That's not what I'd call a conspiracy. It's not a secret that many things continue to be just out of tradition or inertia. I'm sure you can think of a few of them in academia. The fact that actual research was conducted on the subject shows that I'm not the only one wondering. –  Mar 10 '18 at 17:32
  • @neuronet I've clarified. (Glad to see that the HNQ brought all the busibodies :) ) –  Mar 10 '18 at 17:36
  • Regarding blinding, this post is probably of interest to you: http://blogs.plos.org/absolutely-maybe/2017/10/31/the-fractured-logic-of-blinded-peer-review-in-journals/

    As for the benefit of peer review: I haven't dived into it too deeply, but although the benefits of peer review in general don't seem to be that disputed, the "formal" version of peer review as is currently common has had its fair share of academic criticism: https://doi.org/10.12688/f1000research.12037.3

     – Vincent Mar 12 '18 at 11:09
  • 2
    There's quite a bit of debate around this one, and a lot of conflicting or overlapping evidence. I summarised some in an editorial here: http://fossilsandshit.com/wp-content/uploads/2018/02/EON-ISMTE-Jon-Tennant-article.pdf (I'm also the author of one of the papers mentioned above). – Jon Tennant Mar 13 '18 at 14:18
  • Being a bit late for the party, I'd like to note that not all venues that seem to be double-blind are fully so. E.g., in a conference, author names may be released after the initial reviews are submitted, but before the committee takes the final decisions. – Alexey B. Mar 21 '18 at 22:00
  • 1
    @NajibIdrissi Not a "conspiracy" - a "conspiracy theory". I've used wider definition, that is "challenging established practices without any proof". And finding a valid research to prove something is exactly the point of Skeptics SE. Skeptics was established precisely to answer questions like yours. That's by dismissing it as "website for conspiracy theorists" you're dismissing your own question. – Agent_L Apr 26 '18 at 08:49
  • 2
    @Agent_L Thank you for enlightening me regarding your non-standard word uses. –  Apr 26 '18 at 09:03
  • 1
    @NajibIdrissi It's your dismissal of Skeptics that was non-standard. – Agent_L Apr 26 '18 at 09:27
  • 1
    Regarding the premise of this question, see How often can the reviewers correctly guess the identity of the authors when the review is double-blind? – lighthouse keeper Feb 17 '20 at 13:34

4 Answers4

79

Yes, there is scientific research on this topic.

Using data from ACM WSDM'17, A. Tomkins et al. showed that double blind reviewers are less likely than single blind reviewers to accept papers from famous authors, top universities, and top companies.

See: http://www.pnas.org/content/pnas/114/48/12708.full.pdf

Update: As mentioned by @thomas-supports-monica in the comments, there is some evidence that the article linked above may have some issues. See https://papers.nips.cc/paper/8770-on-testing-for-biases-in-peer-review.pdf .

Matteo
  • 2,741
  • 12
  • 22
  • 9
    Does the conclusion that you cite, in itself, show that the system is beneficial? For this does one not also require an assumption that single blind reviewers accept such papers more than they "ought to"? – Rupe Mar 09 '18 at 15:38
  • 4
    @Rupe Insightful question (+1 for logic), but it seems hard to imagine this being the case. – jjc385 Mar 09 '18 at 15:55
  • 1
    "Beneficial" is not well defined. There are so many entities impacted in the acceptance or not of a paper (authors, reviewers, organizers, funding agencies, universities, companies, the scientific community, the general public) that it is likely that what is beneficial for one entity would be detrimental for another. – Matteo Mar 09 '18 at 16:04
  • 27
    @Rupe To expand on other comments, it is assumed that research should be published exclusively based on the quality of its content, hence no effect should be due to the author's names and identities, and the research shows not to be the case in single blind reviews. – Marco Capitani Mar 09 '18 at 16:30
  • 4
    My initial reading of Matteo's answer was that the initial "Yes" was a response to the title. If I'd read it as "yes there is research into the question" then my logical point might not have occurred to me (and I can see that this is a better reading, since there's a comma not a full stop after "yes"). But, sticking with my logical pedantry, I'm not convinced by @Marco's point. Surely this research (as described here, at least) just shows a difference between single- and double-blind? The "effect due to author's names etc" could be there in the double-blind case and not in the single-blind. – Rupe Mar 09 '18 at 16:37
  • Or is it that this research shows that single-blind studies are less likely to favour (rather than merely "accept") papers from famous authors etc? – Rupe Mar 09 '18 at 16:42
  • 4
    @Rupe Actually, if you really want to make the logic here sound, I'd add that the answer doesn't say anything about the overall acceptance rate of double- vs single-blind reviews. Maybe the acceptance rate of double-blind reviews is simply much lower across the board (though it seems to be implied the effect is stronger for famous authors). Caveat: This is almost certainly clarified in the linked reference. – jjc385 Mar 09 '18 at 20:44
  • 11
    @Rupe: You're saying that reviewers might be biased by author names only in the case where the reviewer isn't told the author names? – user2357112 Mar 09 '18 at 21:38
  • 4
    @Rupe How would you test whether the papers from famous (whatever) were accepted more than they ought to be, if not by removing the identifying information and seeing if that changes the acceptance rate? – Ray Mar 09 '18 at 21:43
  • For an answer that's trying to be scientific this is a surprisingly unscientific answer... – user541686 Mar 10 '18 at 00:36
  • 3
    @Matteo After taking a look at the paper, I'd suggest appending the following sentence: "This is despite the fact that double blind reviewers actually accepted more papers overall." (See my previous comment for some reason why.) – jjc385 Mar 10 '18 at 03:08
  • 2
    I agree with @Rupe , it is not obvious that this is a benefit of double blind review. One of the criteria journals use choosing articles for publication is how interesting an article is for readers, and I am not ashamed to admit that, all other factors being equal, an article by a famous scientist is more interesting for me than one by his lesser known colleague. One might say that this is not fair, but, first, the famous scientist probably did something good to earn his/her reputation, second, fairness cannot be the only criterion of publishability. – akhmeteli Mar 10 '18 at 13:38
  • Since I don't have access to the article, I ask here: does it distinguish between reviewers' recommendations and editors' decisions? The latter are never blind. – Martin Argerami Mar 10 '18 at 15:55
  • @MartinArgerami I believe it looked only at reviewer's recommendations ("bids" to accept or reject, and some rating of the quality of the paper). Interesting point about editor's decisions. – jjc385 Mar 10 '18 at 20:18
  • @akhmeteli: "One of the criteria journals use choosing articles for publication is how interesting an article is for readers" - if that is primarily determined based on who wrote the paper, I'd call that a highly questionable approach to start with. As you write: "all other factors being equal, an article by a famous scientist is more interesting for me than one by his lesser known colleague." To establish that all other factors are indeed equal, the authors have to be ignored for the review. The editor can then still select "more interesting authors" among the set of submissions ... – O. R. Mapper Mar 14 '18 at 05:49
  • ... recommended for acceptance by reviewers, but the latter arrive at their recommendation based upon the papers' content alone. Note that reviewers do not usually get to systematically compare submissions. So, their task does not involve determining the relative interest two submissions might generate after establishing that all other factors are equal. – O. R. Mapper Mar 14 '18 at 06:01
  • @O.R.Mapper : "if that is primarily determined based on who wrote the paper, I'd call that a highly questionable approach" - I did not say anything about "primarily" - it' s just one criterion among others. "The editor can then still select "more interesting authors" among the set of submissions" - Sounds reasonable, but life works differently. For example, among the criteria for Nature's reviewers (!) (https://www.nature.com/authors/policies/peer_review.html): "Of extreme importance to scientists in the specific field. Ideally, interesting to researchers in other related disciplines." – akhmeteli Mar 14 '18 at 14:15
  • @akhmeteli: I'd indeed argue that this "extreme importance" should be determined by reviewers without looking at the authors, but with respect to the current state of knowledge in the field. But a separate question here on [academia.se] what exactly is typically meant by "interesting" in those reviewer guidelines could reveal interesting answers. – O. R. Mapper Mar 14 '18 at 14:19
  • @O.R.Mapper (ctnd): It's the same as with authorship of fiction: the author's name sell copies. And scientific publishers are often capitalist enterprises, they want to sell copies. And even if the publisher is a non-profit, they are still interested in circulation. Again, you can say that this is not fair, but see my first comment on fairness. And I would like to add on who should determine if the article is interesting, editors or reviewers: reviewers can have a better idea, maybe that is why "interesting" is among criteria for Nature's reviewers. – akhmeteli Mar 14 '18 at 14:23
  • @akhmeteli: "scientific publishers are often capitalist enterprises, they want to sell copies" - I suppose that brings us back to the question what kind of "beneficial" the OP is asking about. Are they asking about being beneficial for ensuring a high quality of publications? Or are they asking about being beneficial for increasing a publisher's profits? "Again, you can say that this is not fair, but see my first comment on fairness." - which, frankly, I disagree with. "reviewers can have a better idea" - again, "interesting compared to the existing body of knowledge" (reviewers can ... – O. R. Mapper Mar 14 '18 at 14:29
  • ... indicate an opinion about that) and "interesting compared to the popularity of paper authors compared to other submissions" (reviewers cannot tell about that, because they usually do not have the same overview of all current submissions as an editor) are two entirely different things. And for what it's worth, the Nature reviewing guidelines you link to say "All Nature Research journals offer a double-blind peer review option". Hence, either the "interestingness" criterion is silently dropped for those submissions (the guidelines say nothing about that), or referees are ... – O. R. Mapper Mar 14 '18 at 14:30
  • ... expected to being able to also evaluate how "interesting" a given manuscript is without knowing who wrote it. – O. R. Mapper Mar 14 '18 at 14:34
  • @O.R.Mapper : " "extreme importance" should be determined by reviewers without looking at the authors" - I guess this is just your opinion. "But a separate question here" - I'd be interested in the link. "what kind of "beneficial" the OP is asking about" - I just said " it is not obvious that this is a benefit of double blind review" You seem to agree that opinions can differ on this issue. And I guess we disagree on fairness:-). I agree that editors can be better judges of "interesting" in some respects, but I just said that "reviewers CAN have a better idea" (you may add "in other respects" – akhmeteli Mar 15 '18 at 02:27
  • @O.R.Mapper : "Hence, either the "interestingness" criterion is silently dropped for those submissions (the guidelines say nothing about that), or referees are expected to being able to also evaluate how "interesting" a given manuscript is without knowing who wrote it." There is another possibility: "Interesting" is just one of the several criteria, so if an author wants double-blind review, let them have it. You see, I oppose double-blind review because it requires wasting time on anonymizing, but if an author is willing to do that, I have no problem with that. – akhmeteli Mar 15 '18 at 02:40
  • 2
    Unfortunately, the statistical analysis in the paper cited in this answer is flawed. See: Ivan Stelmakh, Nihar B. Shah, Aarti Singh. On Testing for Biases in Peer Review. NeurIPS 2019. – Thomas Feb 02 '20 at 18:10
29

It's always struck me as a rather odd system, with no discernible benefits

Given the ample evidence that non-blinded peer review is biased against a number of different "types" of authors, including women, those with names associated with certain regions, etc. I'd argue that there's a great many discernible benefits. A few examples:

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5360442/ http://science.sciencemag.org/content/sci/333/6045/925.full.pdf

etc. I'd suggest given the widespread evidence for implicit bias in a number of different fields, that the onus is on non-double-blind systems to demonstrate that they're not harmful, rather than vice versa. However, it's also been shown that single-blind reviewers preferentially favor top universities, authors and companies over double-blinded colleagues in this study: http://www.pnas.org/content/114/48/12708

Fomite
  • 51,973
  • 5
  • 115
  • 229
  • 4
    The question is how much double-blind-review really helps. Within my field I can identify the top researchers just by their research topic, used methods, language or how they draw figures. Sometimes even the title is enough to know who did it. And references usually tell you a lot too. –  Mar 09 '18 at 23:19
  • 16
    @DSVA I will note that, given I often review for a double-blind journal, I've conducted an informal evaluation of how often I correctly guess an author when I'm like "Oh, I totally know who this is...". It's below 50%. Your mileage may vary, of course, but I suspect that while people underestimate their implicit bias, they also overestimate their ability to break blinding by content alone. – Fomite Mar 09 '18 at 23:21
  • 3
    @DSVA And how does your guessing work for new authors in your research area, are you just assuming a new name couldn't possibly match the top researchers? A new name that intentionally looks to emulate the best in the field in terms of format, style, and the hot topic because they're trying to break into it. – ttbek Mar 10 '18 at 01:10
  • 1
    While the first part of your answer is interesting (even though I knew it already), it's not an answer, it's a comment. The second part of your answer is unrelated to the first part and already in the other answer (literally the same link). –  Mar 10 '18 at 08:11
  • @Fomite Didn't your informal evaluation entice you to try and guess who the authors were in cases where you wouldn't have tried if you hadn't been evaluating? –  Mar 10 '18 at 13:59
  • @NajibIdrissi No, they were all cases where just reading I've gone "Oh, I totally know who this is..." and then followed up later. – Fomite Mar 11 '18 at 02:37
3

This isn't a scientific article, but offers a different take than the other two answers:

http://blogs.plos.org/absolutely-maybe/2017/10/31/the-fractured-logic-of-blinded-peer-review-in-journals/

moorepants
  • 557
  • 3
  • 8
3

This recent study in finance/economics is relevant to the question and I found it rather striking despite some limitations. It was a field experiment that essentially had 3 conditions:

  1. Anonymous peer review
  2. Non-anonymous peer review, author is Nobel Laureate
  3. Non-anonymous peer review, author is relatively unknown post-doc

The authors listed are real people. Peer reviewers were solicited and given one of those 3 pieces of information about authorship, but in all cases the content of the manuscript was identical.

Fig. 1 in the paper tells the story. For post-doc, 66% of reviewers recommended reject, 25% major revision, 8% minor revision, 2% accept. For Nobel Laureate, 23% recommend reject, 19% major revision, 38% minor revision, 21% accept. In the anonymous case, 48% recommend rejection, 28% major revision, 22% minor revision, 2% accept. Anonymous author is treated modestly more kindly than the non-anonymous postdoc.

As a secondary finding, researchers also varied when the disclosure of author identity happened — either before or after peer reviewers accepted the invitation. Reviewers were a good bit more likely to accept the invitation when they knew they'd be reviewing the work of a Nobel Laureate.

Most obvious weakness of this study in my view is that the authors differ in more ways than the career accomplishments the study's authors emphasize; they also differ in age and race/ethnicity at minimum. For the topic of discussion here, though, it is relatively immaterial why peer reviewers are biased by knowledge of author, just that they are for whatever reason.

Note: pre-print, non-paywalled version available here.

commscho
  • 528
  • 3
  • 5