2

I need to reproduce the simulation done in a research paper I read. I tried to do it by myself last 2 months, but unfortunately without success. I wrote to the first author of this paper but didn't get any reply still. I am afraid he will ignore my email. I am sure he receives a ton of such emails as mine. :( One of my friends suggested me to use help someone external with reproducing the simulation result of the paper: a freelance who will simulate it for me.

I have never used a freelancer service for my research and do not know if it is a suitable solution in my case.

have you ever had such an experience? Do you use freelancers for helping you with simulations?

Field: computer engineering, satellite communication

Aid22
  • 21
  • 1

1 Answers1

4

Reproducibility in science is in crisis in several disciplines. Psychology is one, and while there has been no study out to my knowledge, computer science is another I suspect.

If you tried to reproduce the results for two months and was unable to, there's a real chance that they simply don't work, or work under a very specific set of circumstances.

I argue that if you only need to reproduce another work’s simulation to compare with your own work, you are under no obligation to reproduce it if the authors have not provided the public with the tools to do so.

It is perfectly acceptable to “shame” them and state something like “while X’s work is a natural point of comparison, the authors did not provide publicly available code or data that supports their findings. Thus, we were unable to benchmark against their results”.

If you need the simulation results, and have the funding to hire someone to reproduce them - then absolutely hire someone to help. I would however caution you that there is a real chance that you won't be able to do it since the results are simply not reproducible.

Spark
  • 27,465
  • 10
  • 62
  • 101
  • If it were as easy as "Here is the paper, turn that into code please" OP would probably have been able to do it. That's why I would think a hired low-cost freelancer will not be able to help OP with anything. You could get another CS grad student/knowledgeable person to help you with reproduction and then add them as a co-author. – til_b Jun 13 '23 at 14:49
  • 1
    And +1 for shaming the code non-availability! – til_b Jun 13 '23 at 14:50
  • One describes the algorithms, one provides data, but the code? Are you serious? It is too much to demand this, more so to shame for not providing. It shows your complete misunderstanding of how code development works. Some parts of code can be bound by a commercial license, can run only on specialized hardware, or be written in a exotic programming language. There are many closed source scientific packages that are also used and sold to industry. For instance Mathematica, COMSOL, Gaussian. It is a blessing that these are proprietary codes that can attract excellent scientist to develop them. – yarchik Jun 13 '23 at 20:39
  • 1
    I’ve been involved in some code development projects, so I have some idea how they work. If you’re not offering people a way of reproducing your results, then you’re not doing good science in my opinion. I’m assuming OP is competent, and if so the authors did not offer enough information to reproduce their work - not OP’s problem. Saying they didn’t is the simple truth. – Spark Jun 14 '23 at 00:06
  • 1
    @yarchik I'm quite surprised by your surprise about sharing code in research. Perhaps you think more about the context of industry where company trade secret is more of a big deal? In academia, the majority view now is to push for more open sharing of code. So at the very least, even if you disagree, it shouldn't have come up as a surprise for you that some people hold the view of openness (sharing code). At least that's how it is in computer science field. – justhalf Jun 15 '23 at 06:55
  • @justhalf Think of this, some bright students establish a startup after PhD. They want to monetize their knowledge. Why? Because funding in academia is a finite resource. Now everyone can stay. Think of Stephen Wolfram who established Mathematica. The code is extremely useful for mathematicians, physicists, industries like finances and many more. Or think of Nobel Prize winner John Pople. He established Gaussian, it is still being sold to academic institutions. Why do you demand to open it? No one is going to do such code development in spare time. – yarchik Jun 15 '23 at 13:59
  • @justhalf Scientist have right to copy right their work. See here one extreme case of this https://retractionwatch.com/2023/06/07/journal-pulls-paper-from-ethiopia-for-unlicensed-use-of-questionnaire/ U.S. law encourages academic scientists and their universities to protect and profit from their inventions, including those developed with public funds. https://www.science.org/content/article/pay-or-retract-survey-creators-demands-money-rile-some-health-researchers – yarchik Jun 15 '23 at 14:07
  • Precisely. If you have copyright then you can share it while you're still protected. If it's patentable, then go ahead, patent it before sharing if you want. But sharing is also important, especially given the replicability crisis, having the community able to replicate and reproduce others' work is important, so bogus publications may have higher risk to the authors. – justhalf Jun 15 '23 at 14:31
  • 1
    My answer doesn’t say “share all code always”. It says you cannot have it both ways. If you choose to not share your code for whatever reason, don’t expect people to trust your results or replicate them for you. – Spark Jun 15 '23 at 23:36