What prevents people from manipulating data to write better papers?

Question

Many papers use nonstandard data sets, which means the authors could have manipulated the data behind the scenes to make their paper look more impressive. It seems that this could be done very easily: for example, one could cherry pick the items of the data set to make it seem like an algorithm has 90%+ accuracy while in reality it's about 60%.

What prevents people from doing this? Is there something I'm missing?

Are you asking what kind of safeguards or practices reduce the probability that such junk papers get published? — Antonio Vargas, Apr 05 '17 at 20:21
Sure, you can, and if done correctly it will work at first. But when you are caught, your reputation as a scientist is lost forever. As a wise man said, you can fool all the people some of the time, and some of the people all the time, but you cannot fool all the people all the time. — Federico Poloni, Apr 05 '17 at 20:25
Are you asking about running an experiment, then throwing out experiment data points that makes your idea look bad? Or are you asking about evaluating an idea using an experiment that you know is going to make your idea look good? Or something else? This question is very unclear. — ff524, Apr 05 '17 at 20:35
The content of this question seems reasonable to me (a little naive, but a sensible question), so I assume the downvoters are objecting to the phrasing. I'll try editing it; feel free to revert if my edits don't capture what you intended. — Anonymous Mathematician, Apr 06 '17 at 00:52
In science everything should in principle reproducable. Have you ever thought about what this could imply regarding manipulation of data? I think it means that if you manipulate your data you have to be very lucky to guess right or eventually the fraud will be detected. — NoDataDumpNoContribution, Apr 06 '17 at 06:23
If you're willing to lie and have built up a reputation as a scientist over decades then you can get your papers published in the most prestigious journals. The problem is, that you need to a) Have a reputation to lose in the first place and b) Risk losing it when people can't replicate your experiments. — Valorum, Apr 06 '17 at 08:37
@LeonMeier I wouldn't necessarily disagree, but most "non-reproducible junk" is probably the product of straightforward mistakes rather than fraud. As somebody said in a different context, "never attribute to malice that which can be explained by stupidity". — alephzero, Apr 06 '17 at 09:00
And then someone uses your algorithm on his own data set and finds out it doesn't do what you said it would do. Your reputation is ruined, and you will never publish anything again. — Polygnome, Apr 06 '17 at 10:31
@Polygnome "...you will never publish anything again...." This may be exaggerated. Your reputation may take a dent but often enough flaws in pulications are downplayed as mistakes and subsequently forgotten because the attention has already moved on. I'm quite sure that systematic verification of published (hitherto unchallenged) studies would reveal quite some percentage of unreproducable works, for whatever reasons... — NoDataDumpNoContribution, Apr 06 '17 at 11:06
"What prevents people from doing this?" Maybe most people aren't so bad per se? Maybe they just don't want to. — NoDataDumpNoContribution, Apr 06 '17 at 11:08
@Trilarion Yes, I have simplified the process, but at the core, thats what academia uses to prevent fraud. High entry barrier + harsh consequences. not enough papers are systematically reproduced, though. Thats certainly something that should be improved. — Polygnome, Apr 06 '17 at 11:08
I don't think this is a duplicate to the linked question. The answers here show that the deterrent is not the detection method - discussed and asked for in the linked question - but rather the consequences to career and the lack of a longtime benefit from manipulation. — Grebu, Apr 06 '17 at 11:42

score 37 · Accepted Answer · answered Apr 05 '17 at 21:04

Can I use what I described above to publish a paper without much hassle?

Yes, it's easy to write a fraudulent paper, and it's not hard to get it published if you are willing to publish in a journal with low standards. (Some predatory publishers will publish literally anything if you are willing to pay them.) You might even get it published somewhere respectable if you fake everything convincingly, by getting results that are good enough to be impressive without being so implausible that reviewers become suspicious.

Is there something I am missing?

I'd bet that what you're missing is the role of reputation in academia. If you don't maintain a good reputation, you'll be ignored and marginalized, with no ability to influence the field. If your only goal is to produce a paper, then there's a lot of scope for mischief, but this generally won't lead to a successful research career. If people are unable to build on your work, they'll become suspicious and your reputation will suffer. (And if they aren't even interested in trying to build on your work, then by definition you aren't having an influential career.)

It's not impossible to succeed in academia by fraud, as demonstrated by the people who succeeded for some years before they were caught. However, it's much harder than faking a single paper.

Very nice answer. One could maybe add that more and more journals require/encourage side by side publication of primary data. On the other hand, one needs to be aware that a certain part of scientific publications is faked. I think I remember a systematic too optimistic reporting on new drugs. They are almost always less effective in the long run than reported initially when they are discovered which seems to indicate maybe not outright data manipulation but at least biased interpretation of data. — NoDataDumpNoContribution, Apr 06 '17 at 06:28
@Trilarion Medical research is just crazy in general. A lot of these weird claims that x can lead to y are based on flimsy data, but it makes great headlines so those are the "discoveries" we hear. It's kinda scary. — JMac, Apr 06 '17 at 10:41

score 15 · Answer 2 · answered Apr 06 '17 at 04:48

What prevents people from doing this? Is there something I'm missing?

In addition to the factors mentioned in Anonymous Mathematician's excellent answer, I think another factor at play is that while it is not very difficult for a dishonest person to (temporarily) get ahead by faking data, the rewards for doing it aren't very great compared to the effort it takes to implement this strategy successfully. Even the barrier to entry in academia, getting a PhD, is high enough to deter the most dishonest people, who are the kinds of people who may be tempted to use fraudulent strategies to attain professional success, money etc. And the payoff for this kind of fraud is ultimately quite modest -- you might get a job with tenure, but is that really worth the trouble of working on something you have no real passion for, and doing so in a dishonest way, for so many years?

The bottom line is that Academia is simply not an attractive place for fraudsters, certainly much less attractive than other industries or career tracks (which I won't name) where fraud and other antisocial behavior can reap you much greater rewards in a much shorter period of time. So yes, a hypothetical fraudster who ends up in academia can gain some traction in their career using dishonest methods (until they are caught at least), and occasionally we hear stories about such people. But these stories are rare precisely because academic research as a career option is most appealing to people who by and large are passionate about advancing human knowledge, and therefore are not predisposed to committing major acts of scientific fraud.

score 8 · Answer 3 · edited Apr 06 '17 at 05:59

An example is Jan Hendrik Schön. He manipulated data on a big scale. But after people found out about it, he lost his job (and academic career) and even got stripped of his doctor title.

Apart from this, I guess that most scientists are actually interested in honest academic research (i.e. finding out something truly new) and not in a career build on lies. Moreover, many scientists suffer from some other extreme, they believe that they are imposters (while producing valid science).

What prevents people from manipulating data to write better papers?

3 Answers3