Why are papers accepted even if they don't release code or data to allow reproducibility?

Question

Recently, we had a lecture about Reproducible research on of the slides was:

• Reproducibility: start from the same samples/data, use the same methods, get the same results.

• Replicability: conduct again the experiment with independent samples and/or methods to get confirmatory results.

Replicability = Reproducibility + Conduct experiment again

• Replicability might be challenging in epidemiology (recruit again a cohort) or molecular biology (complex cell manipulation).

• Reproducibility should be a minimum standard. One should strive to at least make his/her own research reproducible.

I find many articles that use software and when they hint what kind of analysis did they don't provide the code nor the data. There are public free ways of storing data and code of studies, and even with a DOI.

So far, I thought that some data might not be freely available because it has some private information (my field is bioinformatics), or the authors intend to use it for further investigations and want to keep for themselves the data.

The same happens with the code is the intellectual property of the lab or principal investigator, but retaining the rights of the code don't goes against the replicability.

Why are these papers accepted and publicized, even if they don't allow reproducibility?

Related: Reproducible Studies? and an example of the problem it causes Can up to 70% of scientific studies not be reproduced?.

Edit:
Some other papers about replicability: 1, 2.

• Excel case: in this paper we can see an example. Reviewers of Growth in a Time of Debt estimated (note that this is not measurable/checkable afirmation) that the analysis could bring the results presented.

  (I couldn't find a description of the methods used for the analysis on the paper, but it is a different field of mine and I have skimmed through).
  But in new methods without prior experience/validation how can one estimate it without looking to the analysis itself?
  And in "old" methods, how bad would be to share them if they are already checked?

• "Understand why replicability is important, and you'll understand which guidelines and rules should be applied, and how to deal with research where guidelines are skipped.":

  Replicability is important because science is about finding objective mesurable relations. This makes the relationship independent of who performs the study. But this can be discussed/answered on another question :)

  I am aware that we do our mistakes, (see my other question here on academia), but we should aim for the best behavior and the best science.

• "Put another way, there are finite resources so the more you rerun the same code the less scientific progress you make overall"

  I don't think that we make less scientific progress overall rerunning the same code. Checking that we know for sure that A is true is far better than work for 3 years or more and then discover that A was wrong.
  How many finite resources are/were used on studies based on those the ALS Therapy Development Institute couldn't replicate?
  At the same time the induced pluripotent cells were hard to reproduce and replicate, but this isn't a software based, or the recent example of @tpg2114, 3 years to replicate their own study in 4 new settings.

• Quality of academic software when sharing it, here it seems that it is better to share the awful, crapy code rather than hide it.

• Necessity of the reviewer of the code and data was answered here. In short:

  Of course, the degree to which a referee is expected to verify the correctness of results varies greatly between fields. But you can always choose a personal standard higher than what's usual in your field. Just realize that good refereeing takes a significant time investment.

It seems that the comments are extending, and from the answers it seems that it is not the job of the reviewers or the journals, it might be a job for the reader (and of no one), or it doesn't worth it to make articles reproducible because it is too hard.

Answer 1

There are multiple answers to your question.

1. Academic publishing assumes good faith. At least in the circles where the majority of people are genuinely interested in the advancement of science. This is the only viable mode of sharing scientific information between adults, although it sometimes fail as we all know it. Publishing should firstly serve the purpose of communicating results in a fast and convenient manner and second provide some sort of quality control.
2. Sharing code and data is not a necessary condition for a study to be reproducible. It surely lowers the threshold of work that has to be done by other groups to reproduce the results, but clear and comprehensive descriptions of the methods and algorithm, as well as how the data was gathered, suffice. In fact, it's even better since starting from scratch will avoid the reproduction of results that are due to artifacts in the implementation or the original data.
3. Acceptance standards vary vastly between journals. Nowadays anything can be "accepted and publicized" providing you pay the "article processing charge". The question is where was it published and by whom. The most prestigious journals in my field have strict guidelines about the study design and sharing of materials and methods, the other ones don't.

Answer 2

Sometimes reproducing results is as much work as producing them. If I give you my code, you not only have to check whether it gives the results I published, but also whether it is correct. Reading poorly written code is a pain, so it might be easier for you to write it in your style again. Also I might have used a language like Fortran or some obscure computer algebra system.

The same is true for certain computations, in particular if they involve a lot of case distinctions, or if there are many different ways to obtain the result.

So I would leave the reproduction to the reader if I have the feeling that my guidance would not be helpful. The problem with this view is of course that it is open to rationalizations: Writing the code in a reader friendly way is too much work, so I invent a reason not to have to do it. Preventing this would be the job of the reviewer, but why should they be less lazy than me?

Answer 3

On top of the existing answers regarding the current state of reproducibility in peer review, it is important to consider that norms regarding reproducibility are rapidly changing in peer review and publication among:

Peer reviewers

The Center for Open Science endorses a standard disclosure request that you can add to any peer review to request that the authors provide minimal information on reproducibility:

I request that the authors add a statement to the paper confirming whether, for all experiments, they have reported all measures, conditions, data exclusions, and how they determined their sample sizes. The authors should, of course, add any additional text to ensure the statement is accurate. This is the standard reviewer disclosure request endorsed by the Center for Open Science [see http://osf.io/hadz3]. I include it in every review.

I use this myself and have found that at least a handful of journal editors will press the authors on this front.

Journal Editors

• Several top journals in psychology are leading the way in terms of incentivizing replication-friendly publications by providing "badges" indicating that the paper has released raw data, had its protocol preregistered, and/or has released study materials. Recent empirical evidence has suggested that such incentives are effective.
• Numerous journals now require a statement about whether data are publicly available, or an explanation of why they cannot be made so (e.g., due to sensitive information). There are too many to list, but here's a commendable example from Science:

After publication, all data and materials necessary to understand, assess, and extend the conclusions of the manuscript must be available to any reader of Science. All computer codes involved in the creation or analysis of data must also be available to any reader of Science. After publication, all reasonable requests for data or materials must be fulfilled.

Authors

Even when submitting to a journal that does not have any requirements regarding reproducibility, some authors are choosing to make this information conspicuous anyway. On my own papers, I now add a "Research Transparency" statement before the Acknowledgments, saying simply:

All raw data, materials, and R code are publicly available at [my Open Science Framework repository URL].

I also say something to this effect in my cover letters when submitting papers. These voluntary disclosures may help promote new community norms, hopefully eventually making poor disclosure a red flag rather than the status quo. In fact, I experienced a small example of this when a reviewer recently commented:

I appreciate the authors [sic] releasing all their data and protocols online.

Answer 4

It's often technically difficult to recreate the same runtime as another person used, so the code may not do the same thing on your machine. In practice, we're usually not talking about tested, portable, production quality software. You may need a certain operating system, certain libraries to be installed, the correct compiler or interpreter version etc.

Reproducible computation is something the SciPy community was deeply interested in a few years ago, and seemed close to getting a handle on. I'm not sure how far they've come since (I'm not a scientist, so it was off my radar), but I know it was an important issue for IPython Dev, and Continuum Analytics were working on it as well.

You need to share the whole setup, maybe as a machine image for Amazon Web Services, or use Docker or something like that. You basically do your work in VMs, then share whatever people need to reproduce the VM.