16

Recently, our group is trying to reproduce the result reported by a paper whose authors are from an Ivy league university. We are unable to reproduce the results because there are several implementation details are not mentioned in the paper. Hence, we decide to write the first author, who is now already a faculty member in another university, an email.

At first, we asked for the code, which we consider is perfectly fine, as the paper has been published. We feel the code in a published paper is no longer a secret in this transparent research era where reproducibility is highly valued. However, he simply ignored our email (3+ weeks, no response).

Then, we thought, OK, seems that he is reluctant to share the code, so let's just ask him to clarify several implementation details so that we can implement the thing ourselves and hopefully, we can reproduce the result. So, we sent a second email which very clearly asks for clarifications. Again, he ignored the email (1.5 weeks+, no response).

We now feel angry and start suspecting the authenticity of their reported results. However, we cannot accuse them of anything, since we are not able to prove that they cheated, which would be a felony if they really did.

We always feel that upon the publication of a paper, its authors, or at least the correspondence author, hold responsible for any inquiries regarding the paper, especially when the authenticity is being doubted. What they chose to do - ignoring our email - is really irresponsible.

What can we do?

Disclaimer

Thanks for the answers and comments! It is interesting that many start besieging me on my "bad" attitude in the email.

Just to clarify,

  1. I wrote perfectly polite emails to the correspondence author;
  2. I have NOT questioned his results or whatsoever.
eykanal
  • 48,364
  • 18
  • 119
  • 216
Sibbs Gambling
  • 5,841
  • 8
  • 45
  • 69
  • 25
    ... which would be a felony. — Say what? – Mad Jack Jul 09 '14 at 18:30
  • 6
    It might take a new faculty member much more than a month to respond, especially right at the end of term. – StrongBad Jul 09 '14 at 18:45
  • 9
    Two things: if you asked in the past three months you have to realize that May is extremely busy with the end if the semester, then many academics go on break for at least part of the summer. Second, if you have a bad attitude (judging from your rush to judgment about felonious behavior), people are apt to ignore you. – RoboKaren Jul 09 '14 at 18:51
  • 20
    The authors have no obligation whatsoever to respond to your emails, nor to share their code. – Cape Code Jul 09 '14 at 19:15
  • 1
    @Jigg Maybe true for the code sharing part. But if they are not obliged to email-clarify the unclear parts in the paper, why is there such a thing called correspondence author? – Sibbs Gambling Jul 09 '14 at 19:24
  • ... to email-clarify the unclear parts. I have never received or send such an email. Usually a paper stands on its own and reading it thoroughly should be enough to answer any conceptual answers on it. On the other hand, no one will clarify engineering details of implementation, because this is not the original purpose of the paper – Alexandros Jul 09 '14 at 19:37
  • 1
    @Alexandros Not so, if the details are important parameters of the algorithm. – Sibbs Gambling Jul 09 '14 at 19:55
  • 6
    I didn't say you couldn't ask, people occasionally do. I did it once ore twice to request original data to compare it to mine. What I'm saying is that there is no obligation to do so. – Cape Code Jul 09 '14 at 20:05
  • 5
    @Alexandros You seem to be saying that no academic will ever answer any question about their papers. After all, any question must either be about the central ideas of the paper (and you seem to be saying that such a question is redundant because every paper describes its central ideas in perfect clarity) or not about the central ideas (in which case, you seem to be saying that it should be dismissed as irrelevant and not worthy of an answer). In my experience, academics are usually willing to answer reasonable questions about their papers. Often, this kicks off research collaborations. – David Richerby Jul 10 '14 at 00:38
  • 2
    Have you tried phoning? Or is it just these two emails? – starsplusplus Jul 10 '14 at 02:17
  • 11
    @FarticlePilter they are definitely not obliged to email-clarify the unclear parts in the paper - most gladly do so, but it would also be perfectly acceptable and ethical to ignore your requests if they want to do so for whatever reasons. Corresponding author is responsible for the correspondence with the journal editor during the acceptance and review process; but there is no duty or obligation to correspond with random readers, unless they want to. – Peteris Jul 10 '14 at 02:38
  • 4
    It's not okay to expect people to share code... codes in our research area are regularly licensed for millions of dollars, half of which goes to the university. We would never be permitted by the legal department to distribute any of it. – tpg2114 Jul 10 '14 at 04:26
  • 1
    @FarticlePilter As long as there won't be open science, you'll have to live with it... – Franck Dernoncourt Jul 10 '14 at 05:53
  • 2
    What exactly does the school's sports league have to do with its research?? –  Jul 10 '14 at 06:36
  • 3
    @Alexandros As you said, the paper usually stands on its own, but it does happen that some details get omitted, or things are ambiguous. I get that we should't jump to asking authors as soon as we come to a hard spot in the paper, but the whole point of publishing your results is that they should be reproducible and verifiable (if they're not, they're essentially worthless to the scientific community). I'm just wondering, what would you suggest to do if an procedure detail is missing which is important for reproducing the experiment and thus, the results? – penelope Jul 10 '14 at 11:56
  • 2
    @penelope ...results is that they should be reproducible and verifiable. I agree. But how many times have you received an email asking clarification questions about your paper? How many times you have asked for such details? As Jigg says it is not bad to ask, but it is not that common (maybe it is and I am mistaken) nor it is usual practice. Otherwise, the author of a paper with 1000 citations would be a support center. – Alexandros Jul 10 '14 at 12:32
  • 6
    "We feel the code in a published paper is no longer a secret in this transparent research era where reproducibility is highly valued." - I can think of a lot of code of mine that is neither secret nor in any shape to be shared. If someone asks me for that code, I generally decline, because a non-negligible amount of my time would be required to pack all the necessary files and assemble instructions on how to compile and/or run the respective programs. When in research, I write my code for demonstrating a point, not for redistribution, and that's exactly what it is optimized for. – O. R. Mapper Jul 10 '14 at 12:57
  • 2
    @Alexandros I'm a PhD student at the beginning of my career. But, in two distinct cases where a) a detail crucial for implementation was missing from the original publication or b) I just couldn't wrap my mind around a concept in the paper, both times my supervisor suggested to contact the authors. Note that these were relatively short and simple questions, but important. I get that there is no obligations from the authors to resolve my b) case, (or the a) case), but I'm still wondering what is the "policy" on case a). I assumed, it is at least somewhat common since adviser suggested it. – penelope Jul 10 '14 at 12:57
  • 1
    @ChrisWhite That gives a rough indication that the school is not bad. Don't nitpick, please. – Sibbs Gambling Jul 10 '14 at 13:01
  • Please take extended discussion to [chat]. Some superfluous comments (those edited into the question) have been removed. – eykanal Jul 10 '14 at 14:04
  • 1
    @MadJack I am not making any accusations, but I feel from the tone of the question that there is a belief that the original authors fabricated their results. If that is true and their research was federally funded, then there very well may be a felony. I am also open to the possibility that the OP simply can not reproduce the results because of skills deficiency. – emory Jul 10 '14 at 17:41
  • 1
    If they are federally funded, then you can FOIA them. They are obligated by law to respond and they will be hounded until they respond. I would try more traditional methods first. – emory Jul 10 '14 at 17:44
  • 4
    Why not just write one of the other authors? – JeffE Jul 11 '14 at 07:15

3 Answers3

30

How do you know he ignored the email?

Maybe he never received it because it was filtered before he had a chance to see it. Maybe he hasn't read his email this month because he's on vacation. Maybe that email address was good when the paper is published but not good now, but it's also not bouncing.

If one channel of communication doesn't work, try a different one. Call his office phone or send a letter to him. Write to a different author saying you've been trying to reach the corresponding author without success and you want to check the email address. Don't assume anything when you have no information.

brian d foy
  • 380
  • 2
  • 7
  • 4
    +1 for "call his office phone". A phone calls is a very effective way to make a first contact. Then you can follow up with an e-mail summarizing what you understood from the conversation and asking for clarification of anything you didn't understand. – Christopher Bottoms Jul 10 '14 at 13:41
24

We always feel that upon the publication of a paper, its authors, or at least the correspondence author, hold responsible for any inquiries regarding the paper, especially when the authenticity is being doubted.

Did you actually state that you do not believe in the results of the paper? If yes, this is disrespectful and plain rude. And most people ignore rude remarks from strangers in the internet. When requesting the help of any other human being, you should be polite and cautious. Also, in Academia you must be very careful when you refer to someone's work. Especially his PUBLISHED work. Because that means that the scientific community has already accepted his claims and you are the one who must prove that his results are wrong and not the other way around (if you ever get published on this subject which be hard to do without his help). So, acknowledge the fact that he has nothing to prove and he will be doing you a favor if he accepts to share his code.

Also, think of the possibility that he does not want to share his code. It is his code after all (and not public domain) and he still has the right to keep it for his personal use. He may also plan to expand on his work and sharing the code prematurely deprives him of the 3 months - 1 year time-advantage over you, since you still have to implement it yourself before expanding on the current state-of-the-art, i.e., his work. In this case, provide him with an alternative. Say that you are willing to send him your datasets and if he agrees to do the experiments for you and report to you his results, it should be good enough for 90% of the cases and everyone is happy. You have the necessary data to compare against your method and he did not have to share his code, which is a logical compromise.

Also, academia is a place that you need to use your social skills. You need collaborators and not enemies / antagonists. In that sense, ask for help politely and expect that NO is a very possible answer on the other party. Also, if he is an established researcher and you are not (perhaps you are famous too - I do not know) there is the case that he ignored the email, because he simply does not even know who you are, what you do and how you will use his code. Usually, telling little things about yourself in the introductory email, sending a link to your personal homepage and google scholar profile, suffices not to consider you a crank and reply to you.

Also, sometimes the first author is a graduate student and the student might not want to share his code because he feels threatened. So, check all the authors profiles. See who is the most senior in the paper and CC him as well in your emails. In that case, the senior professor might encourage the student to share his code despite his objections. Either way, it cannot do you any harm.

As you see, there are multiple reasons why he did not reply to your email. Also a little flattery works on most of the cases. Note that in a sense you find this work fascinating because otherwise you would not struggle to improve it. It is not bad to say so and usually this kind of politeness opens more doors and is more useful in the long term.

Alexandros
  • 18,921
  • 3
  • 49
  • 84
  • 15
    Brilliant. The only quibble I have is that journal acceptance doesn't mean that the scientific community has accepted the data and conclusions as accepted, merely that they have passed the initial scrutiny of peer review (thus likely to be true). Replication and the test of time is still necessary. – RoboKaren Jul 09 '14 at 20:05
  • 3
    Thanks @RoboKaren. You are also right for "replication and the test of time is still necessary". – Alexandros Jul 09 '14 at 20:09
  • +1 and thanks a lot for the suggestions. "CCing the senior professor" is an action worth trying. – Sibbs Gambling Jul 09 '14 at 20:19
5

I just want to address one minor point:

At first, we asked for the code, which we consider is perfectly fine, as the paper has been published. We feel the code in a published paper is no longer a secret in this transparent research era where reproducibility is highly valued.

It does not matter what your feelings are but rather what is the policy of the journal the article was published. Some journals require disclosure of data or source code but others do not. Also if the research was funded by the NSF there maybe mandatory disclosures policy. I suggest you to see if this is the case.

Dr Pangloss
  • 860
  • 9
  • 16