4

In computational research, authors of publications are often required to open-source their code. However, while having open-sourced one's code improves chance for journal/conference acceptance, reviewers in general do not read code (for obvious reasons, as reading code is very time consuming). As a result, code are usually poorly written / badly documented.

If I'm unable to understand/run a codebase (provided that I'm quite familiar with the language/framework it is using) even under good intentions, and my own implementation fails to reproduce its results, who's responsible?

This is a real concern since, as students/researchers, our time is a scarce resource and we all have a life outside academia. I don't want to spend an unlimited time on a codebase that I cannot understand.

Adam Wilson
  • 354
  • 1
  • 10
  • 5
    If you can prove the correctness of your own, perhaps that will solve the problem without having to prove the incorrectness of theirs – Luke Sawczak Dec 20 '21 at 13:47
  • 1
    Have you contacted the authors and provided a bug report including error messages or seemingly incorrect results that you obtained? – Brian Borchers Dec 21 '21 at 23:22
  • At first I thought this question was opinion-based, but then I remembered why peer review exists. – Anonymous Physicist Dec 22 '21 at 01:12
  • @LukeSawczak That sounds like a good idea. Open-sourcing my own well-documented implementation, even if it doesn't work as claimed, allows others to identify potential problems. – Adam Wilson Dec 22 '21 at 03:45
  • @AnonymousPhysicist I'm not sure if I get what you mean. If I'm interpreting correctly, I agree that open-sourcing code is just like saying: "look review I've done it" but it doesn't hold any more value than that. – Adam Wilson Dec 22 '21 at 03:47
  • @BrianBorchers That's a good idea, and I've encountered authors that respond timely. Having said that, a lot of open-source codebases are poorly maintained, and few even claim that they are academic labs, not software companies, and are hence not responsible for answering questions. – Adam Wilson Dec 22 '21 at 03:48

3 Answers3

2

If you cannot reproduce a result as a student, then that can mean two things: the code is wrong or you are wrong. Both are likely: mistakes happen and students are students because they don't know everything they need to know (otherwise they would not need to be students). Authors are obviously not responsible for a lack of skills of students, but they are responsible for their own mistakes.

What the consequences are, or should be, are two completely different questions. If you are responsible, but there are no consequences, then ...

Maarten Buis
  • 43,487
  • 8
  • 87
  • 152
-1

No, you aren't responsible for reproducing the results of others. You are responsible for producing good and accurate results. Reproducing the results of others is good if, but only if, those results are correct.

It is possible that the code you found has led to an incorrect result. The code, per se, isn't the issue, but the results.

If your code produces different results, they may be correct or not and your own code may be good, or not. But if you want to know why there is a divergence a well designed set of tests might tell you exactly why. They may tell you where the results diverge and if you can use that to call the paper you've found in to question then you have the basis for a publication.

The diversion, of course, might also show you where your own code went wrong.

And, even if you don't have code yourself, you can use such a set of tests to verify the validity of the found code.

But, at the end of the day, it is hard to trust results produced by crufty code.

Buffy
  • 363,966
  • 84
  • 956
  • 1,406
-1

... reviewers in general do not read code ... fails to reproduce its results, who's responsible?

Peer reviewers and editors are collectively responsible for ensuring published work includes enough information that the work can be reproduced. If the code is deterministic, already published, and it does not reproduce, that is a failure of the reviewers and editors.

If the research in question is statistical in nature, it's normal for small samples not to reproduce. Only physics is responsible.

If the typical PhD student in the field does not have the expertise to reproduce the research, it has been poorly presented and the peer reviewers should have pointed that out.

If the code does not reproduce because the PhD student does not have typical PhD student skills, then the faculty who should have trained the PhD student are responsible.

My advice: peer review better, when it's your turn to review.

Anonymous Physicist
  • 98,828
  • 24
  • 203
  • 351