0

I found a potential wrong application of machine learning validation methods in one paper recently published in Nature Energy, which is the best energy journal (>50 impact factors). The authors used k-fold cross validation over a forecasting model on gasoline demand under COVID-19 with Google mobility time-series data and the COVID-19 data. They claimed they don't have over-fitting issue, as both training and testing cases reach R-square above 0.8. That should be wrong, as time-series validation should be considered instead of K-fold cross validation. It would be cheating to use K-fold in this case. Please correct me if I am wrong.

I reached the editor, and there is further evaluation ongoing since then. However, the editor does not promise to take the proper editorial action since I am not willing to disclose my personal information. Is that true? Could anyone make suggestion on this? How to report this misconduct in a correct way if I want to remain anonymous? Thanks a lot for your advice!

RSJ
  • 47
  • 1
    Is what true? – Buffy Jul 28 '20 at 20:54
  • 15
    How do you know this is misconduct? That is, an intentional effort to deceive. – Nathan S. Jul 28 '20 at 21:04
  • 3
    Perhaps the technical aspect could be queried (with neutral phrasing) over on Cross Validated. – Jon Custer Jul 28 '20 at 23:07
  • 2
    Why not try PubPeer? – ZeroTheHero Jul 29 '20 at 00:19
  • 1
    (Note: not a hard-core statistics person) Looking at descriptions of k-fold and time series, I don't see why one could not apply the k-fold methodology to a time series - one just has to partition the data appropriately. You would really need to rigorously support any accusation of "misconduct". – Jon Custer Jul 29 '20 at 01:27
  • 1
    @Buffy I suppose you are trying to be helpful? But I think it is clear enough they are asking whether that is a normal valid response, and whether it is usual for editors to not take action without OP providing personal info. – sleepy Jul 29 '20 at 04:58
  • 3
    "Misconduct" is not an appropriate term in this scenario. Misconduct is very serious and could ruin careers. But the more important question I have is are those authors known to you? I hope this whole episode is not coming out of some sort of rivalry. If you do not know them then why not send the authors an email seeking clarification? Check if they are replying you and then figure out the next steps. – kosmos Jul 29 '20 at 08:48
  • See also the question https://academia.stackexchange.com/q/157911/80985 – Gnudiff Oct 22 '20 at 09:13

3 Answers3

10

Two things here:

Firstly, if you want people to take you seriously, you really should use your real name. I quote:

Use your real name. Not using your real name indicates that you are trying to avoid suffering any potential negative consequence of your claim being incorrect. Using your real name indicates that you are sure enough to be ready to suffer potential negative professional consequences if you are mistaken, so you can be taken more seriously. If you are not completely sure about your claim do not waste her time.

If you must be anonymous I don't think you have many good options. Some (many?) people simply aren't going to take you seriously. You would have to convince someone who is willing to use their real name to do the convincing for you.

Secondly, there is an important difference between misconduct and an error. An error is simply an error. It is a mistake, it isn't intentional, it can be corrected. Misconduct goes much further and claims that the authors intentionally attempted to deceive their readers. You deal with both differently. With errors, you do one thing. With misconduct, you do something else. Your description of the problem sounds like an error, not misconduct. Do remember that a misconduct allegation is more serious than an error. Everyone makes mistakes, but misconduct can lead to job terminations or awarded degrees being rescinded.

Based on your description you aren't alleging misconduct, you are alleging a mistake. Be clear which one you're alleging because they are different things. Confusing something like this does not bode well for being taken seriously either.

Allure
  • 127,528
  • 50
  • 325
  • 493
  • 3
    I don't think it is that simple. People are afraid of repercussions, not simply of tarnishing their own reputation by making a mistake. – Andrés E. Caicedo Jul 29 '20 at 01:56
  • I strongly believe it must be possible to report misconduct anonymously and be taken seriously. Too often reporting misconduct is a career-limiting move and as result it doesn't get reported. –  Jul 30 '20 at 06:32
5

In contrast to what the others have said, I believe there are reasons to want to be anonymous when reporting misconduct.

However, what you describe is not misconduct, but just bad interpretations of their data. Many, many papers use incorrect methods, or faulty interpretations. As long as this is not a deliberate attempt to mislead, it wouldn't be classed as misconduct.

While I understand that pointing out flaws in other peoples work is scary, particularly if they are more powerful than you, I think there is less of a case for anonymity. Of course in peer review you are often anonymous to the author, but you are not anonymous to the editor and it is understandable for the editor to want to understand who you are to judge who to believe - you or the author.

henning
  • 35,032
  • 10
  • 121
  • 151
Ian Sudbery
  • 38,074
  • 2
  • 86
  • 132
3

I won't comment at all on the validity of your criticism, because that is off-topic for Academia.SE. I'll only answer the more general question, though I will assume that the conduct is not really misconduct (as mentioned in a comment) but rather an argument about proper statistical approaches.

If you're not willing to attach your name to criticism, you put little weight behind it.

I'm not sure this choice rises quite to the level of a retraction, which would be done through the editor. A letter to the editor may be appropriate, which could be published in the journal (often after inviting a response from the original authors). If so, I'd suggest you convince a co-author to write the letter with you, preferably someone with a stable position (tenure), both to give it more credence and to give you a bit of protection (and also to verify your concerns).

These days, there is an alternative venue through social media where you may be able to raise an issue anonymously, but unless you already have a strong following it's probably difficult to get it picked up. Perhaps you could contact someone in the field who has a history of calling out errors in statistical approaches - I won't recommend anyone specifically but there are several blogs and Twitter accounts devoted to exactly this.

Bryan Krause
  • 114,149
  • 27
  • 331
  • 420
  • "If you're not willing to attach your name to criticism, you put little weight behind it." Why? If an error is identified, it is identified. It is an error. Regardless of whether it was John, Mary or someone else who first identified it. – Andrés E. Caicedo Jul 28 '20 at 23:06
  • 6
    @AndrésE.Caicedo I didn't say it makes it not an error, I said it puts little weight behind it. In this case you have N=? authors putting their stamp of approval on the paper, plus probably 2-3 peer reviewers, all who have given their names to the editor. The editor may not be an expert on this particular statistical issue, and OP seems to be making a pretty serious allegation yet won't even put their name on it (and they are sufficiently unsure to ask here). For them, it's something like 6 voices they trust vs one they don't know. – Bryan Krause Jul 28 '20 at 23:10