How does “reproducing other labs' results” work?

Question

I don’t understand where "reproducing results from other labs" fits into the running of a given research lab.

(I'm not in academia, and I confess I only consider this function in the context where it's News due to fails - ie, "So & So breakthrough is published, but other labs were not able to reproduce …". Nor do I recall a popular science headline about a discovery, that bothers mentioning successfully reproduced results.)

Such news mention might imply a quiet ongoing process.

With limited time & resources, it seems researchers would stick to their own project, however adjacent new results may be. Rather than pause their work for a tangent, all to say "Yes, we found those same results", or "No we didn't". I don't see their benefit.

Or is "reproducing outside results" simply built into all labs' work flow, and how so?

Obviously its a needed function, but I've never heard how it's actually structured. (quid pro quo - I'll reproduce yours if you'll reproduce mine.. ?)

Answer 1

With limited time & resources, it seems researchers would stick to their own project, however adjacent new results may be.

Well, yep, there's the problem. Most labs have funding to perform a new project. Hence, most results are never attempted to be replicated, until it becomes of interest to someone, or easily achievable on the way to doing something "novel."

Occasionally, students, like undergrads or master's students might perform a replication as their thesis, where there is less expectation of originality.

Obviously its a needed function, but I've never heard how it's actually structured.

It's not.

Answer 2

The other answers are so far covering situations where some result is reproduced because of doubts. But there are also situations where the published result allows (is the only or presumably best known way) the other lab to obtain material they directly need for their projects. Reproducing the original paper is then no waste at all* but a by-product of using the results of the paper for other research:

• Think chemical synthesis: a synthesis method or process is published which allows making substance or material X or a particular group or type of substances. Someone needs X for their synthesis of Y, and thus they start reproducing the paper, so as to actually have material/substance X.

• Think of measurement methods: a paper describing new instrumentation, and then another lab realizing that the measurement problem they encounter in their work would be better met/solved/improved by the published setup.

• Think of data analysis algorithms: someone publishes an algorithm to do X, with application example being the well-known benchmark data set Y (or publish code for X together with data set Y). Others realizes that the algorithm may be useful with their application problem, too. Typically, when familiarizing themselves with the algorithm, they'll start with Y, too, and thus reproduce the original paper.

---

* I'd like to emphasize that reproducing other findings is often not wasteful: there is actually indication in several fields that we'd need to do far more of this. But: for certain types of results there is low/no incentive to do this (or it's even outright discouraged because funding agencies and publications have novelty requirements)

Answer 3

There are a number of possibilities. Usually they are triggered because someone knowledgable has some doubt about published results. That might be just intuition, or it might be based on earlier work that seems inconsistent. Sometimes the author(s) are mistrusted for one reason or another, such as being employed by someone with a vested interest in certain "results".

A scientific paper will include its methodology. The usual process of checking a result is to use that same methodology with new data, perhaps a larger data set (or not), and see if the results match. Statistical studies, for example, might be like that.

Sometimes a result is based on an experiment, say in chemistry, described in the paper. The "checker" will run the same experiment in their own lab and see if the results match.

Sometimes a result is based on software and a dataset. The check could use the same software, when available, with a new data set or independently developed software built to the same specifications. Independent development guards (somewhat) against software errors.

However, the most valuable sort (IMO) is when someone takes the same research question and develops a different methodology to check the research claim and gathers what evidence is available to see if the results agree. This rules out (hopefully) some results too determined by a given methodology. For an interesting example of what can go wrong see the Stanford Prison Experiment.

Any of the above are possible, along with others.

Yes, researchers want to run their own experiments, but they also want valid results to make it to the mainstream as future work will be based on it. This is especially important in some fields, such as health, and medical research. Casting doubt or disproving a published result can be a very valuable contribution to knowledge.

Answer 4

If you publish results, then at some point you would hope that someone picks it up and bases something better on your research.

So you published that if you do A, B, and C, then X happens. And I thought of a way to do Y and Z if X happens. But it doesn’t work.

At some point I suspect your results are not correct. And I start a project trying to replicate your research. Or not. Because showing your results are wrong is a good contribution to science.

Answer 5

Question author says the following:

I'm not in academia, and I confess I only consider this function in the context where it's News due to fails - ie, "So & So breakthrough is published, but other labs were not able to reproduce...". Nor do I recall a popular science headline about a discovery, that bothers mentioning successfully reproduced results.

In addition to the other answers, I'll try to provide some (potentially) instructive/informative examples.

Reproducibility is the hallmark and cornerstone of science!

What separates science from other forms of explanations of the world around us and for example the healing arts is reproducibility. If I explain to you how to do something in detail, all my techniques and precautions, and what results I get along with my estimated errors, then you should be able to follow my instructions and get the same result, within some range of variation that is consistent with recognized estimates of experimental error.

If I say that mixing blueberry juice and thumbtacks makes plutonium and tell you how I did it, then you should be able to try to make plutonium too. If it doesn't work for you, my results are irreproducible, and there's a journal for that.

Here are a few extreme examples that may bracket how the reproducibility (or lack thereof) of a given result was or was not confirmed in a timely way.

Alzheimers - Tragically, NOT enough effort was spent trying to verify a result, time & lives wasted?

Biology SE's Does the recent concern over several papers about Aβ*56 call into question the association of Alzheimers Disease with any amyloyd beta oligomer forms? gives more details on a really tragic failure of science to police itself.

Not enough effort was spent trying to reproduce an experiment pointing to a certain form of amyloid protein's link to Alzheimer's disease. Instead many biologists spent years doing not-so-fruitful research based on the premise that this result was correct and several pharmaceutical companies spent (essentially our¹) money and (their) time developing drugs and doing drug study after drug study that did not demonstrate results.

The scientific community failed us in this case because they did not confirm that the original results were reproducible.

Several papers were retracted years later, some by the journal without the authors' input. There was plenty of damage to go around, but no way to recover the time and money misspent.

GSI positron peaks

The positron-electron peak puzzle: results from APEX (Ahmed et al (1997) Z. Phys. A 358, 235–236 (1997)) summarizes a few million dollar project at Argonne National Laboratory to try to verify some strange results seen for years by two competing groups at GSI Puzzling Positron Peaks Appear in Heavy‐Ion Collisions at GSI (Physics Today v. 38, no. 11, p. 17 (1985)) There was no room for such peaks in modern understanding of physics, they stubbornly remained at the few standard deviation level and release of specific details of analysis (and the raw) data was never completely forthcoming.

It was irritating to nuclear and particle physicists, so a clean start on a different continent by a group with no ties to GSI was initiated. A huge amount of effort went into trying to reproduce the results, including consultations with those groups to make sure nothing was omitted.

Cold fusion

This is another "irritating" result for which there was no room in Physics for the phenomenon, but the potential payoff was incredible.

Unlike the GSI positron peaks, this one was relatively easy to approximately reproduce - you could stick some palladium in water, run a current through it and put a neutron detector next to it or put it in a calorimeter and measure the heat produced compared to electrical power, but in this case due to greed the originators withheld certain specifics that made honest attempts at reproducing the results unable to implement tests that were exact reproduction.

Lack of sufficient information to test the reproducibility fed the storm and the pockets of some individuals.

---

¹Research budgets of established pharmaceutical companies comes from revenue - i.e. how much we pay directly or indirectly for the medicines that keep us alive and healthy. This also means time and money was NOT spent on potentially more helpful drug development.

Answer 6

I have some divergence with the existing answers; my viewpoint is that reproduction is naturally built into the scheme of researchwork. That is to say, it is neither occasioned by suspecting some results, nor is it something that has to be additionally incorporated into workflow.

This answer is based off an engineering discipline; I find it to be valid for experimental as well as simulation groups.

Research in such fields is often incremental, which implies that each bit of research is built on some previous research and adds a small advance. In other words, previously published research becomes a starting point. It is then imperative and natural to reproduce the previous work, so that you have a known starting point. Without doing so, you would not know whether your results are consistent or not.

In materials engineering, a perennial effort is to make materials stronger. One group develops a new class of materials and shows that it is stronger than the existing materials. Another group wishes to push the envelope further and introduces some processing method which will further strengthen the material. They need to first produce and test the starting material to benchmark their results. In doing so, they are reproducing/validating the first group's results.

Or is "reproducing outside results" simply built into all labs' work flow, and how so?

It is, provided that there is some connectivity between research of different labs. In fields where each lab is producing fundamental results, I suppose this is not built in.

(quid pro quo - I'll reproduce yours if you'll reproduce mine.. ?)

No quid pro quo; it works just as citations work. If your work is important and relevant, if it prompts further research, then someone will reproduce it as part of a further study.

In my field, we informally use this as a way to test the response; if something is not reproduced in a few years, it's probably not relevant enough to pursue much further.

Answer 7

I want to emphasize the part that the answers that cbeleites and AppliedAcademic provided, repoduction is very often a core parts of the scientific workflow. But that part is much less obvious in the literature as it is often not explicitly published.

Any time you start a new project, odds are that you're working on top of existing results. Either the subject of your investigation, e.g. a specific protein, chemical or organism has already been studied or the methods you're using are well established. And the first step is to establish that your lab setup works how it should, and this is easiest if you only check one thing at a time.

So for example if you want to do experiments on a specific protein you need to produce it first. And once you did that you need to make sure it behaves like expected, and for that part you would usually compare it with previously published result. So you would check the structure (or rather some experiment that works like a fingerprint for the structure like an NMR HSQC) against the literature or check the function of the protein if it is an enzyme. To do that you would typically reproduce the literature, to make sure your system behaves as expected. And of course this is a bit more complex in detail, you might be interested in specific behaviour or properties of protein X under conditions Y when you add Z. There are plenty of things that can go wrong, so trying to do some experiments first where you know the expected results is usually a good idea.

The other case is when someone publishes a new method. Those are often published with data on some kind of model system first, those model systems usually are well behaved and easy to handle. If you try out that new method first you typically would do that on the same model system, to make sure it works and you're doing everything correctly.