Are experiments described in scientific papers *actually* peer-validated today?

Question

Background: the answers and comments on this question prompted this question.

I know that when a new scientific theory is published there is a rush for people to devise experiment to prove or disprove it. However there is lot of research that involves original experiments that are not necessarily related to new not-yet-proven/accepted theories.

Now, obviously science relies heavily on peer-verification of theories and experiments, but are experiments today actually verified by independent peers?

The thread I linked to made me grow some doubts: if an experiment has so much trouble being funded, how is it possible to scientifically verify the results of already published data, especially of high-cost experiments. If there is no incentive in doing that verification, how can articles describing a new experiment be validated scientifically?

Disclaimer: sorry for the possible dumb question, but I had only a brief experience in academia at the beginning of my career (and it was in SW engineering, where "experiments" are way less common than proof of concepts software, so verification is a far less difficult subject there).

As a further explanation of my doubts, the last time I remember an experiment being rejected because it couldn't be validated was the thing about cold fusion by Fleischman and Pons, back in the late 80s. At least that was a case that reached mainstream media. I guess such events (both validation and rejection) are more common in the specialized journals, but I don't remember any other "big event" that reached the mainstream media since then.

Please, note well that I'm not claiming that experimental results never get published by the mainstream media, here I'm focusing on the validation by peers that are independent from the original authors/scientists.

---

Maybe I wasn't too clear, possibly because I didn't use the right terminology.

I'm not particularly interested in the review process details (although I welcome any correction about my lack of terminology), but in actual scientific validation of experimental results.

What I really care for is whether or not published scientific experimental results are effectively put under experimental scrutiny by the scientific community.

Answer 1

There are two related but separate processes in play here.

On the one hand, papers are peer reviewed, i.e., when you submit a paper describing your experiment, results and conclusions to a conference or journal, two to three peers will review it. They review the paper - they typically don't have the time, funding or other requirements to actually re-do your experiment. They will check whether your arguments make sense, whether the experiment you describe can in principle be used to address your research question, maybe run a small simulation.

On the other hand, people will try to replicate your experiment, i.e., run a similar experiment and see whether they get the same results. (Typically they will run a modification, because straight-up replications are very hard to publish, and publications are important to academics.) If you have an exciting result, many people will try to replicate and extend it. If then most of these people cannot replicate your results (take a look at the "replication crisis in psychology"), the excitement of your results will wane.

Answer 2

When laypeople are introduced to the idea of replication, they imagine scientists constantly redoing every paper that comes out, to "check it". At least I did. This is not the case in most mainstream sciences. The closest thing is when a paper is basically announcing the release of a software - then conceivably a lot of people might give it a try, since it's usually easy (or is it?) to just install and run a software on your computer.

This does not mean there is no replication. To be sure, there is a replication crisis, in the sense that we have too many studies that should be replicable but aren't. But it's important to remember that a lot of science can and is replicated.

The replication does not take the form of a researcher going to their PI, and submitting a 20-page document titled "Proposal for Comprehensive Replication of Jones et al. 2022". Replication is rarely a discrete project, it is embedded in novel research. The rare exception is extremely high profile and controversial work - for example, the Yamanaka induced pluripotency protocol inspired many outright reproduction attempts.

A lot of science these days is not "from scratch", but building upon and refining previous studies. Say Smith 2019 discovered that red M&Ms cure cancer in mice. Murphy 2024 might then test different chemical extracts of red M&Ms across various mutant mice, and discover that a specific chemical is responsible for the effect by acting on a certain gene - such a study might often begin with a "sanity check" by repeating a simple version of the original Smith 2019 experiment. Indeed the Smith 2019 regimen may serve as a baseline or control for evaluating the additional claims of Murphy 2024.

Another common situation is when published results are assumed to be true, and an experiment is set up such that if the assumption is false the experiment will produce nonsense results. For example, imagine that Dr. Murphy decides to feed some mice only the filling of red M&Ms, while other mice are fed only the shell. It's expected that either group will either be cured of cancer or not. But if all mice die of chemical poisoning, that obviously raises some questions about the results claimed in Smith 2019. It is often possible, with clever experimental design, to "cover" validation of past results as well as probe the actual hypothesis of interest.

You also mention cost. However, you make the false assumption that reproduction always costs as much as the original experiment. Think of it this way - if you decided to sail from Europe to America, would you need to be an equal to Columbus? A big sink of money, and time, therefore money, in original research is trial and error. There are a lot of unknowns about what experiment would work, so it costs a lot to perform it. However, reproducers can skip all that - with the benefit of hindsight they can avoid the false starts.

Answer 3

Others have discussed the official process.

Generically, the thing is, the process is messy. There is no megalithic authority presiding over science, nor academia in general. We don't have a "king of science." There is no "pope of academia."

As such, it is possible for a false result to remain unchallenged in the literature for some time. This can be due to honest mistakes or actual fraud or various combinations.

The process of science eventually turns to re-examine old ideas. People are hungry for publication. There are teams and groups and milling armies of people hungry for publication. They search through old journal articles, read them, sit and think "I could re-do that experiment in a couple weeks with stuff in storage at the lab." And many variations on this. They re-do the arithmetic in published articles. They request the raw data. They compare other experimental results reported at other labs.

So, at any given moment, it is quite likely the literature contains many errors, mistakes, and a few (hopefully a very small number) of actual frauds. And science will try to correct them.

As Dara O'Briain said: Science knows it does not know everything. Otherwise it would stop.

Another thing that tends to happen is that a result becomes irrelevant due to advances that move us past it. As a trivial example that nobody probably cares about: Suppose somebody claimed they could make electronic vacuum tubes to perform a certain way, but that their result was wrong. The advent of transistors moved us away from vacuum tubes, so nobody cares anymore about the performance of tubes. Possibly such an incorrect result from 60 years ago might hang around in the journals and never get challenged. So there is a category of knowledge that one should take extra caution about accepting as authoritative, because the process of science dropped the topic.

So you can recognize a healthy science endeavour by the presence of people questioning the results. They need to be doing it in a well based manner, not simply sticking their fingers in their ears and going "la la la." They need to understand the issues, the theories, the current state of things. And then they need to be free to ask questions and make critiques. If there is nobody questioning a result that is a warning that the science has gone moribund.

Answer 4

but are experiments today actually verified by independent peers?

In some (rare) cases, yes.

---

The examples that I have are in Computer Science, where there is a big push lately for articles to share their "artifacts", which are environments that allow to reproduce their results.

Major associations such as the IEEE and the ACM now have their own "badges" (IEEE badge and ACM badges) that are awarded only if the artifact is publicly available, if the results can be reproduced, and if the environment is "re-usable" (read, it's easy to tweak their plat-form to test another hypothesis).

Major conferences such as Discotec or POPL now have their "artifact evaluation committee" whose main goal is to check that the results from the papers can be reproduced.

Answer 5

No, peer-reviewers usually just read the manuscript. However with a lot of experience reviewers can often tell from the presented results and the methods description if the research is valid.

If researchers have doubts about the published results they usually do a replication study and then write a letter to the editor.

Answer 6

I might be worth thinking about science from a Kuhn-ian point of view, rather than a Popperian one.

Start with the somewhat surprising assumption that all scientific conclusions are really models of how the world works, and that all models will eventually be shown to be false to some degree.

In Kuhn's model scientists conduct studies under the assumptions of the current Paradigm (i.e. the set of models that people assume to be true). As long as the results of those studies produce data that makes sense under the current paradigm, they support its truth/usefulness of the models that make up the current paradigm. To the extent that those studies propose new models, they become part of the paradigm. Thus we build up "sandcastles", each new theory based on the validty of the those that came before it, and the success of the new thoeries support the validity of the old. But over time, anomolous results accumulate that can't be explained by the current paradigm. At first these can just be ignored, or explained away, but eventually the weight of anomolies becomes so heavy that the models on which the whole paradigm collapses and you have a scientific revolution, or paradigm-shift. The things that survive this shift will continue to form the foundations on which the new sandcastle is built, and overtime will be compressed to form sandstone.

As well as happening at a macro-level (relativity replacing newtonian physics for eg), it also happens at a micro-level. If a paper is published proposing X and providing an experiment that supports it, rather than trying to replicate those results, someone will say "well, if X is true, then perhaps Y is also true", and they will conduct an experiment assuming X is true. If those results are positive, they lend support to X, if they fail, then the probably don't in one go disprove X, but they might start to add question marks to it. If there are enough such results, people will start looking round for other theories to based their investigations on. This usually doesn't happen in a big bang (like it would do at the macro-level), but mostly just people drifting away from believe X - its not demosntrated false, it just that those that don't base their science on it tend to be more successful.

One downside of this process is that it is slow, and at any one time only those with a deep knowledge of the whole field will really have a good idea of what the field believes to be true or not at any given time. In fact, at any given time it would be very hard for someone to say "here is a list of things that people in field A believe to be definately true". We will only really know what was a good call and what not decades or even centuries into the future.

This is of course no good when your results are needed for practical purposes (like treating diseaes). A couple of things to observe here is that even if we can't say for certain that X is true, if you had to bet your life on X or not X, you'd be better off betting on X, even if you only have a qualified belief in it being true. The second is that the medical and health fields are fields were replications really do happen. There are often multiple clinical trials into the effect of a particular exposure or class of drugs. These are generally collected together in meta-analyses which collate all to evidence for a single question to see what the evidence is in aggregate.