30

I found a significant error in a paper with 1000 citations. My advisor agreed that it is a significant error, but discouraged me from telling the author.

It occurred to me that maybe other graduate students also found the error, and never told the author, or they did tell the author and he just never corrected it. I think an author would have very little incentive to correct his own work, especially if they were famous for that work, and readers are strongly incentivized against writing a correction paper, because the author would be mad at them.

How often does this sort of thing happen? Can we really trust the literature?

ff524
  • 108,934
  • 49
  • 421
  • 474
user56416
  • 301
  • 3
  • 3
  • 4
    See What should you do if you spotted a non-trivial error in a highly cited paper?, How to deal with errors in well established papers – ff524 Jun 09 '16 at 21:58
  • 6
    Did your advisor give a reason for not wanting to contact the author? – Kimball Jun 09 '16 at 22:09
  • 1
    @Kimball He just said it wasn't a high priority and that maybe we could bring it up next time we saw him (which would probably be never). I assumed he didn't want to lose the friendship or something. – user56416 Jun 09 '16 at 22:15
  • 10
    This surely happens all the time. And no, you cannot trust the literature. I don't think anyone would seriously claim that you can. Neither authors nor reviewers are infallible. If the result is important, you have to check it yourself; or weigh the effort needed against the risk of it turning out to be wrong. – Nate Eldredge Jun 09 '16 at 22:37
  • 21
    There are a lot of errors where technically what is written is incorrect, but an experienced reader who is carefully studying the paper would notice the error and be able to recognize what should have been written instead. People often don't bother to correct errors of this kind: not so much to avoid embarrassment, more just because it's a big hassle. It may be that your advisor thinks that's the case here - maybe it is obvious to her what is meant, even if it isn't to you. – Nate Eldredge Jun 09 '16 at 22:44
  • 2
    It would help if you could indicate the field of the paper - different fields have different kinds of "mistakes". – Oswald Veblen Jun 10 '16 at 00:34
  • When I was a post-doc, I came across an error in a paper by some very famous researchers. One of the intermediate claims in the paper was clearly wrong. In fact, it was so clearly wrong that I assumed that they authors had merely made a mistake in how they were phrasing the claim, and that the main arguments ought to be correct. Although I knew one of the authors fairly well, I didn't contact him about it. However, the paper was part of a lengthy back-and-forth between two sets of authors, and about a year later, the other group published a strong rebuttal, based, in part, on that error. – Buzz Jun 10 '16 at 01:03
  • 4
    @NateEldredge if indeed it's a "forgivable error", I think it's a good teaching moment for the advisor to then explain to the student why that is. And so it's reasonable for the student to ask for some clarification. – Suresh Jun 10 '16 at 05:23
  • It does happen. During my PhD, I replicated the work in a modelling paper analytically and then numerically, and both times demonstrated that there was an error in the original well-cited paper. My advisor insisted that I must have made a mistake myself but wasn't interested in checking my work.I hadn't made a mistake: the error was there, but I didn't feel I could do anything about it. In my later career, I met the author, but still didn't say anything because I would have needed to redo my calculations to demonstrate it and it was too much work. – Significance Jun 10 '16 at 05:48
  • 1
    True, scientific literature is littered with papers that have errors in them. This article raises concern over the issue of uncorrected errors in scientific literature: http://retractionwatch.com/2016/05/19/retractions-arent-enough-why-science-has-bigger-problems/ – Kakoli Majumder Jun 10 '16 at 12:24
  • Closely related: http://academia.stackexchange.com/a/48228/20058 – Massimo Ortolano Jul 06 '16 at 20:45
  • 2
    How is this question possibly answerable as posed? – mako Jul 06 '16 at 22:01
  • 1
    If it is "a significant error," then presumable it appears in a central argument. Hence, the central argument is flawed. You could present a paper that highlights the issue and gives a valid argument. You could ask the original author to co-author your paper. – user2768 Jul 12 '16 at 12:38
  • @user2768 That is much more difficult than you would suggest. Sometimes the errors are very subtle and extremely difficult to nail down. This is one of the subtle and major issues of the cutting edge of modern mathematics. There are threads on MO asking "what is the error in this paper" in some form or another, and there are rarely any definitive answers. Just heuristic insinuations, or appeals to alternative proofs/disproofs. None of them can point at the exact issue in the exact paper and go "Here, this is the problem." Just "this seems like it is questionable/much harder than suggested" – zibadawa timmy Jul 13 '16 at 01:05
  • @zibadawatimmy, user56416 claims to already know the error. In any case, pinning down the details is surely a part of mathematics. – user2768 Jul 13 '16 at 11:11
  • 2
    @user2768 Easier said than done sometimes. Voevodksy had a multi-year spate when Simpson published a counterexample in 1998 to a theorem Voevodsky and Kapranov had published in 1991. But Simpson couldn't point out what the flaw was. And nobody could find a flaw in the counterexample. It wasn't until 2013, 22 years after he published, that Voevodsky conceded his result was false. I'm not sure if anyone knows exactly where things went wrong to this day. – zibadawa timmy Jul 18 '16 at 04:49
  • @zibadawatimmy, now I understand your point. Although a counterexample suffices to show that a result is wrong, it does not show where the result went wrong (i.e., no particular hole/mistake in the proof is identified). For user56416, presumably knowing the error means knowledge of sufficient evidence (counter-example or otherwise) to show there's a problem with the original work. – user2768 Jul 18 '16 at 09:28

4 Answers4

8

I'm not sure what you are looking for. I know for sure that the seminal paper that made one of my bosses famous had glaring mistakes in it. Despite the fact that his result was completely wrong, the paper started a new research field and is now highly cited (around 1700, I think). No one ever bothered to point out the calculation mistakes, without which the paper would have never gotten accepted, because his proposal got confirmed experimentally. More detailed calculations done by others showed his mistake, but confirmed his intuition. In any case, I can't find survey data in my field dealing with un(der)reported mistakes in papers. But, as you do research, you are bound to find quite a few.

On the other hand, I found this oncology paper on unreported mistakes in oncology papers http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3031354/

Looks like less than 25% of those who spotted mistakes in the papers, actually went on to report them.

The practice in my field is to write the authors about the possible mistake. If they are willing to correct it, there is not much point in escalating. If not, you can write a comment on how wrong the paper was, post it on arxiv and send it to the journal editor. This, assuming you're sure they made a mistake. Most people aren't willing to go through all this pain, but I've seen this many times.

6

Researchers/authors are human.

Humans make errors.

Thus Researchers/authors make errors (if we assume transitivity is given).

So, can we trust literature? Well, being an agnostic I tend to never blindly trust any statement. I found dozens of errors in the lectures I attended and made the experience that lecturers respond very different to criticism. There are some that feel attacked, or those that simply do not bother, but there are also those that will really think about your criticism and will correct you or themselves after having carefully checked the issue in question again.

I think, that a researcher's duty should be to allow and work with criticism as this will lead to an even better understanding of the topic for all involved sides. Also, your criticism shows interest which is actually a good thing and should be rated as such.

A good criticism should focus on the research itself and leave out personal-related stuff.

So my conclusion is, that you shouldn't just do nothing because of those strange unspoken rules society has built up to underline the higher status of academic people. Remaining silent for social fears is a bad thing that leads to a vicious cycle and harms research in my opinion. You should be free to question whenever you feel like it is necessary to question. Consider for yourself for when it makes sense to consume someone else's time. If you are not sure about your criticism, make some own research. If you are sure your criticism is right, if you ask me, for research itself this is the best one can do.

Arno
  • 43,962
  • 8
  • 123
  • 169
kaiya
  • 283
  • 2
  • 9
2

Slightly tangential (doesn't answer "can we trust the literature?"), but:

On several occasions, especially when working with a class of students to work through a paper line-by-line and replicate its results, I/we have found errors — nothing that changes the qualitative conclusions of the paper, but definitely large enough to be confusing to students. These kinds of errors are overlooked by most readers because they aren't going through the paper as thoroughly, but setting the record straight can save a lot of time and trouble for future readers — especially students, who won't have the courage of their convictions and will think that they must have made a mistake/be misunderstanding something.

It helps of course that (1) I was already reasonably well established in the field and knew/was known to the authors and (2) the authors were reasonable people who saw the value in correcting their minor (but confusing) errors.

For example:

Yates A, Stark J, Klein N, Antia R, Callard R (2008) Correction: Understanding the Slow Depletion of Memory CD4+ T Cells in HIV Infection. PLoS Med 5(1): e11. https://doi.org/10.1371/journal.pmed.0050011

In another case, where there was a typo in a single equation that didn't propagate downstream, and the paper was older (>10 years from publication), the author confirmed the error but said (and I agreed) that publishing a correction probably wasn't worth the trouble.

Ben Bolker
  • 4,873
  • 1
  • 20
  • 32
1

Can we really trust the literature?

Let me point out here that "can a paper be trusted" and "all of the calculations in the paper are exactly correct" are not necessarily related. This is something that people get wrong about research, and in particular single papers, all the time. Academic papers are useful dialogue. They are not necessarily 100% accurate or unimpeachable.

Peer review is designed to catch methodological errors. It confirms that the authors followed sound scientific principles as appropriate to their field. Most peer review does not check for errors in calculation, and typically could not even if they wanted because the peer reviewer does not have access to all of the paper's data. However, peer review does confirm that a group of experts all read the paper and agreed that the methods and conclusion make sense.

A paper published in a good venue is not strong because it is guaranteed to be 100% accurate, it's strong because the expert community of reviewers thinks it is accurate. I would assume, though I can't cite any evidence, that academic papers at good venues have a much lower error rate than similar documents that do not go through a rigorous vetting process.

As to whether this error should be reported- it really depends. If this is a purely theoretical result, if it is empirically in doubt, if it substantially changes the result of the paper, then the error should really likely be reported. If the calculation is wrong but the intuition and the model are solid, and there is strong empirical evidence for the model, then the calculation error probably isn't that meaningful.

Here's a practical yardstick: if reporting the error substantially advances the scientific conversation, then it is a publishable result that should be published, just like any other publishable result. If reporting the error is just fixing errata and not substantially contributing to the body of knowledge, then it's not publishable and probably nobody cares.

David
  • 8,266
  • 19
  • 35