1

This question coupled with the general acceptance that a lot of scientific fields have a reproducability crisis has made me wonder whether there should be a specific journal/database that is dedicated to publishing reproductions of existing work.

I understand that this is somewhat drab and unglamorous, but it could serve a few, very useful purposes.

My thoughts on this are:

  • New researchers often reproduce prior results as part of their training. This could give them an opportunity to get some experience in writing to a publishable standard.
  • We could build a database of evidence that either supports or refutes existing findings publicly. A lot of work that finds the status quo is right goes unpublished so there's a whole corpus of evidence to support certain fields that goes wasted.
  • It would potentially serve as a very useful early-warning that something has gone wrong when nobody can reproduce certain results or there are inconsistent results on reproduction attempts.
  • Journals are typically concerned with "novelty" so don't really serve as a useful avenue for this sort of work.

Am I overthinking the potential of this or do people think such a thing could be useful?

ScottishTapWater
  • 2,247
  • 9
  • 23
  • Well, there was the Journal of Irreproducible Results... – Jon Custer May 15 '23 at 16:43
  • 1
    Trying not to be too blunt: People have spent substantial parts of their careers thinking about how best to support replication and reproducibility in the sciences. Your question sounds like you've just arrived at the beginning of these thought processes but have not spent time actually figuring out what others have already done in this area. – Wolfgang Bangerth May 15 '23 at 17:24
  • I think this could be improved by clarifying what your actual question is here. – user438383 May 15 '23 at 19:31
  • 1
    @WolfgangBangerth - Well you're not wrong... Although isn't asking a question on here exactly part of "figuring out what others have done in this area"? – ScottishTapWater May 16 '23 at 09:57
  • 1
    One of such journals is ReScience C "that targets computational research and encourages the explicit replication of already published research, promoting new and open-source implementations in order to ensure that the original research is reproducible." – Olexandr Konovalov May 16 '23 at 20:48
  • @ScottishTapWater But you're not asking a question for people to point you in the right direction. You're telling others "isn't what I just came up with a great idea?" The thing is that others have had these sorts of ideas a long time ago, and have actually done some work to investigate if they could work out. – Wolfgang Bangerth May 17 '23 at 02:52
  • @WolfgangBangerth - No, I was asking what people thought. That includes both positive and negative. I also got a useful answer so I really don't see what your problem is – ScottishTapWater May 17 '23 at 09:11
  • You might want to read what Victoria Stodden has written, for example: https://www.stodden.net/ There's also a report by the National Academies on Reproducible Science. Both of this isn't specifically about reproducing science, but about making science reproducible. – Wolfgang Bangerth May 17 '23 at 17:35

1 Answers1

3

The term used more often is "replication" rather than "reproduction"**.

Journals already exist with the goal you recommend, and other journals explicitly state they accept submissions of replications (though for some I wonder if their editors and reviewers are actually so welcoming as the journals would like to represent).

Specific journal recommendations are off-topic on Academia.SE, but you could search something like https://www.google.com/search?q=journal+replication+studies to find either journals that have specific policies encouraging replication or exist entirely to publish replications.

** - Often a distinction is made between replication and reproduction. Reproducing results means taking an existing data set and coming to the same conclusion as the authors. "Reproducible research" goals would include authors providing data and code that was used to generate their analyses. A paper is reproducible if you can say "yes, their data actually show what they say they show when you perform these analysis steps." Replication, on the other hand, means addressing the same research question with new data: re-doing the whole experiment from scratch or using a new source of data.

Bryan Krause
  • 114,149
  • 27
  • 331
  • 420
  • I didn't realise such things already existed, that's good to know. Also, thanks for the clarification on terminology, I think both are vital. In my line of research I've had constant issues with not having the source code to go along with papers about simulations. So I have 1) no way of knowing if things were actually implemented correctly and 2) no easy way to build on that research – ScottishTapWater May 15 '23 at 16:58
  • @ScottishTapWater Yeah, also standards may vary by field a bit so those terms may not necessarily mean the same thing to everyone. For example, I'm sure there are some cases in CS where running even the exact same code on a different machine (either with different hardware or different supporting software like OS) would be considered something closer to replication than reproduction, whereas in most other places it's not expected to matter. – Bryan Krause May 15 '23 at 17:06