17

When something is peer reviewed, how much trust are the reviewers saying you should give it?

I'm not talking about whether peer review works, I'm asking what peer review means ideally.

For example, I know you should trust a peer reviewed article more than, say, a blog post. At the same time, if an article passes peer-review, that doesn't mean the reviewers are saying that the article should be accepted as eternal truth and never questioned further.

So how much trust does peer review give?

  • Should one trust that citations are claiming what is in the articles they cite?
  • If there isn't anyone who has disagreed yet, should I assume it's true for the purpose of discussions, without further examination?
  • Should public policy be based on it, without further examination?
  • Should I trust data to have been collected correctly?
  • Should I trust data to have been processed correctly?

This probably varies by field (for example, I imagine pure mathematics and related fields, which already have high standards for rigour, are endowed with a ton of trust once peer reviewed).

Christopher King
  • 359
  • 1
  • 8
  • 12
    I imagine pure mathematics [...] are endowed with a ton of trust once peer reviewed I don't think that's true or should be true... I'm only a new PhD student in pure math, and I've read peer-reviewed material with completely wrong (without any doubt possible) things in them. Last week read a peer-reviewed paper where an article cited a 1,000-page book claiming that the book proved a result the author needed -- the book didn't mention the result or the objects mentioned in the result in any way whatsoever... It was pure bluff on the author's part. –  Jan 18 '16 at 16:35
  • 6
    Two popular sayings come to mind "cum grano salis" and "trust, but verify". – Ric Jan 18 '16 at 16:36
  • Somehow related: http://academia.stackexchange.com/q/48224 – Massimo Ortolano Jan 18 '16 at 16:49
  • @NajibIdrissi okay, I wasn't sure. – Christopher King Jan 18 '16 at 16:50
  • 2
    It means that someone supposedly knowledgeable looked over it and found it original, correct and relevant. The reputation of the journal is to some extent at stake if it is not (however, sometimes good journals can be quite badly off), as the review is anonymous. Ultimate acceptance is only via the test of time, and only for discoveries which enjoy a certain modicum of interest. Where not, mistakes may be hidden for long (but nobody cares, either). – Captain Emacs Jan 18 '16 at 16:57
  • 4
    I think everything on your list is the responsibility of the authors, not the peer reviewers. – GEdgar Jan 18 '16 at 17:04
  • 1
    @NajibIdrissi "It was pure bluff on the author's part." That's a very serious accusation, for which you've presented almost no evidence. First, the result may appear in the book in a form that you weren't expecting or it may be an obvious consequence of something that is there: that's very common and something that a new PhD student might not find. Second, the author may simply have been mistaken and cited the book from memory when the correct citation was to some other book. That you immediately assume dishonesty on the author's part is very worrying. – David Richerby Jan 18 '16 at 18:11
  • 2
    @DavidRicherby The result the author mentions is about something called a widget, a completely standard name for it. The word "widget" does not even appear in the cited book. The widget was introduced by Smith. Smith isn't cited in the book. The use of the widget was refined (in the context where the result is needed) by A and B, and then by C and D. Neither A, B, C, or D, are cited in the book. And so on. The widget is a special instance of mogrification – the word "mogrification" doesn't appear in the book. Dishonesty? I don't know. But it's not the first paper of this author that I've read. –  Jan 18 '16 at 18:26
  • 1
    [cont.] (In case it's not obvious, "widget", "Smith", "mogrification" and so on are just placeholder words in the previous comment.) Of course I didn't present any evidence, I'm not about to name the author or the work... I don't think it's the place for this. –  Jan 18 '16 at 18:34
  • 1
    The same question was asked on skeptics meta: http://meta.skeptics.stackexchange.com/q/2125 – Kimball Jan 18 '16 at 18:47
  • 2
    @CaptainEmacs I'd phrase it more "someone supposedly knowledgeable looked over it and didn't notice anything unoriginal, incorrect or irrelevant" - That is, peer review is supposed to catch the bad stuff, but can't guarantee the quality of what makes it through. Reviewers are only human, after all. – R.M. Jan 18 '16 at 21:28
  • @R.M. you get both false negatives and false positives, sometimes both on the same paper. Sometimes, actually, you get really useful reviews, but it is rarer than one would wish. – Captain Emacs Jan 18 '16 at 22:01

4 Answers4

20

  • Should one trust that citations are claiming what is in the articles they cite?

    No. Too many people don't read their sources carefully enough (or even at all).

  • If there isn't anyone who has disagreed yet, should I assume it's true for the purpose of discussions, without further examination?

    There is a wide range of possible levels of trust between something I read in a blog post and something on the level that "the square root of 2 is irrational". I'd put a random peer-reviewed article somewhere between the two extremes, depending on the discipline, the journal, the state of the art and so forth.

    "Without further examination" should not be part of a scientist's vocabulary. Except maybe for the square root of 2.

  • Should public policy be based on it, without further examination?

    No. In particularly not if the article reports on experimental findings in psychology, economics, sociology, medicine and so forth. Most published research findings are false. These disciplines always need replications and meta analyses, because they cannot perform experiments as tightly controlled as, say, in physics.

  • Should I trust data to have been collected correctly?

    I do my best, but I wouldn't trust myself 100% to have collected my data correctly for published work. Stuff always happens.

    In addition, peer review doesn't really enter into this question. Peer reviewers cannot easily assess your data collection - only your description of it. You could have made horrendous errors in good faith, and the reviewer wouldn't know.

  • Should I trust data to have been processed correctly?

    See above. Would you trust software to be bug-free? You shouldn't. And again, reviewers don't review your data analysis as such - usually, your analysis scripts are not part of the bundle you submit.

Bottom line: peer review will increase my level of trust in an article, but not infinitely.

In addition, like a good Bayesian, I trust more surprising (which have a better chance of appearing in the more prestigious journals) findings less. Therefore, I usually expect articles in Nature and Science to be less easily replicable than less "sexy" findings published in other venues.

Community
  • 1
Stephan Kolassa
  • 34,479
  • 11
  • 114
  • 185
5

Should one trust that citations are claiming what is in the articles they cite?

Simkin and Roychowdhury (2003) estimated that authors read only 20% of the works they cite. You cannot even assume that the cited articles exist:

The Most Influential Paper Gerard Salton Never Wrote https://www.ideals.illinois.edu/handle/2142/1697

Should public policy be based on it, without further examination?

No.

Why Most Published Research Findings Are False http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124

Should I trust data to have been collected/processed correctly?

Hauser was a leading scientist in his field and he was making up the data:

https://en.wikipedia.org/wiki/Marc_Hauser#Scientific_misconduct

God help you if Excel was used:

http://lemire.me/blog/2013/04/24/you-probably-shouldnt-use-a-spreadsheet-for-important-work/

Daniel Lemire
  • 311
  • 1
  • 4
2

I see peer reviewed articles as a discussion. A discussion with some rules to discourage people from advancing untested or ridiculous arguments, rules that require one to acknowledge previous papers and present evidence about the new idea, but not much more than that. Published letters are rarely a summary of the community's shared conclusion about something. They are simply a fresh argument or an interesting interpretation of some new idea/data/evidence.

(The longer review articles are often an attempt to form such a shared conclusion, but even they may have some political slant to them.)

Should one trust that citations are claiming what is in the articles they cite?

Usually but not absolutely. I have seen a very believable citation in a front page Nature publication that turned out not to mean what the authors asserted it did. Naturally I only noticed this when I required the information for my own work.

If there isn't anyone who has disagreed yet, should I assume it's true for the purpose of discussions, without further examination?

Should public policy be based on it, without further examination?

In both cases, never.

Should I trust data to have been collected correctly?

Should I trust data to have been processed correctly?

It depends for what you require those data. Data processing and collecting are often done by the least trained and most stressed people in the laboratory, who are sometimes under pressure to cut corners to get something their boss or advisor likes. There is often a great deal of cherry picking in the hard science disciplines to show the data in an exciting way, so to justify a top tier publication.

If the data are going to be supporting your safety case for the airworthiness of a new passenger aircraft, you probably should get the data again and talk to the original authors (in person) about the experiment. If the data are merely supporting a limb of your hypothesis in a new paper, and are widely in agreement with other data published independently, then it is probably sufficient to accept them as they are.

Calchas
  • 1,412
  • 10
  • 11
1
  • Should one trust that citations are claiming what is in the articles they cite? Yes. A competent reviewer knows all major works in the area and would not allow to replace expected references by something little known and questionable.
  • If there isn't anyone who has disagreed yet, should I assume it's true for the purpose of discussions, without further examination? This would not be scientific. Even if there is an experimental work proving something, these experiments are later repeated in other laboratories.
  • Should public policy be based on it, without further examination? A public policy should be based on consensus between multiple peer reviewed articles, taking into consideration where they disagree.
  • Should I trust data to have been collected correctly? Should I trust data to have been processed correctly? No, you need to wait till the experiments will be repeated in multiple independent laboratories.
algorithmic_fungus
  • 3,791
  • 15
  • 24