15

I found a major flaw in reasoning in a psychological experiment, but I am not even related to psychology, I just attended to that course at my university in my spare time. The flaw I found is strictly logical, and also quite tricky, so I think it is feasible that no one noticed it yet.

This mistake (if it turns out to be an actual flaw) has huge consequences since a few studies validated the results (by repeatedly committing this mistake) and many psychological sub-fields referenced the study.

I want someone to validate my concerns about that study, and then if possible I want to discover the consequences of that error, and make humble suggestion on how to deal with the situation. If my notice would be published anywhere, I want to make sure my name will be seen in that article. (It would be awesome)

What would be the way to go in my situation if such things can be published at all?

Little Edit: The error is highly controversial but my point of view should be true. It is similar to the Monty Hall problem.

Irwin
  • 8,847
  • 4
  • 33
  • 41
Jani Kovacs
  • 251
  • 1
  • 4
  • 8
    I'd venture that you're maybe jumping the gun a little, by worrying about the publication. I think it's more important to check if the bug you found is indeed a bug. In my area of research, there are many people who claim to have found bugs in major results but haven't really. – Suresh Apr 11 '13 at 02:43
  • 6
    Be cautious in how you present yourself as well. You do not want to come across as someone from another field trying to show people in psychology how it's done. –  Apr 11 '13 at 02:56
  • 2
    Generally speaking, a publication (of whatever nature) devoted to pointing out a flaw in somebody's work is not done in academia, as far as I've seen. What you can do is publish something constructive (an alternative approach, an interesting counterexample, etc.) and leave your verdict implicit. For direct criticism, use unpublished communication (e.g. peer reviews). – reinierpost Apr 11 '13 at 08:16

2 Answers2

28

In order for this to happen, you need two things:

  1. You need to communicate with specialists and convince them that you have identified a logical flaw. You might worry that they will take your idea without giving you credit, but you shouldn't worry too much about that: the more worrisome scenario is that neither of you will be able to convince the other and it will end with a stalemate. Without expert advice, it's unlikely that you can present your idea in a way that would be publishable, so talking with experts should be the first step. (Of course unlikely is different from impossible, but it's best to plan on getting advice.)

  2. Once they understand you, the experts need to care. Many experiments have loopholes and caveats that are well known to experts but not described in popular accounts or even introductory college courses. You may end up hearing "Yes, that's logically possible, but it's not a possibility we think is important." Or "Yes, Jones and Smith pointed that out in a paper last year, but I still think the conclusions are true and the experiment is illuminating." Using an argument that is not 100% logically air-tight is not in itself a problem outside of pure mathematics.

I'd suggest starting by talking with someone from the course you took (the professor or a teaching assistant). If all goes well, either you'll convince them that this is new and serious, or they'll convince you that it isn't.

Anonymous Mathematician
  • 132,532
  • 17
  • 374
  • 531
  • Thank you for your answer, especially for pointing out, that the expert will need to care, if I am right. But when(if) my observation turns out to be correct and significant(even questioning the conclusion) is it automatic that I will receive an opportunity to publish, even suggesting a way to interpret the consequences? note:the prof is highly respected in the subject – Jani Kovacs Apr 11 '13 at 00:36
  • 1
    If others agree that the observation is original and important enough to publish, then you should be an author and play a substantive role in writing the paper. (Other people might become authors too, if they contribute to understanding the observation and its consequences, and you would have to discuss and agree on what to write. However, it would be unethical for them to take your observation and publish it without including you.) – Anonymous Mathematician Apr 11 '13 at 01:32
  • 7
    is it automatic that I will receive — No, of course not! – JeffE Apr 11 '13 at 03:57
  • 7
    It's great advice to talk to an expert. I know that my mathematician husband often has views on flaws of psychology experiments which, while technically correct, actually wouldn't change the value of the experiment for psychology if corrected. In other words, they are often things that do not relate to the part of the conclusion that is interesting to the psychologist. – Ana Apr 11 '13 at 13:33
11

I think there is no way for you to publish this without understanding the psychology:

  • You won't know the significance of your work.

  • You will receive rebuttals that require you to have domain-specific knowledge. (It might even be that your argument does not hold due to some domain-specific reason.)

You can discuss it with experts, but you need to know what it is you are actually claiming.

As a personal example, I spent the last 2+ years, along with my co-author, writing a paper where we discredit the typical statistical analysis used in network motifs. Writing this paper changed me as a person.

I feel there would have been no way to write this paper without understanding the biological side (my research was in pure mathematics and computer science). Time and time again, I would be asked questions that required knowledge from biology, and I don't know or I don't understand the question are not appropriate responses.

Nowadays, I would say biology (or, at least, computational biology) is a field I study.

Douglas S. Stones
  • 1,149
  • 1
  • 10
  • 16
  • Can we get a citation for your paper on motifs (if already published)? I am highly interested since it is relevant to my research. – Zenon Apr 11 '13 at 19:26
  • It's currently under review. I can send you the submitted version if you would like. – Douglas S. Stones Apr 11 '13 at 19:33
  • That would be absolutely awesome! I am sending you now an email (I don't like leaving too obvious traces of my aliases on the internets). – Zenon Apr 11 '13 at 20:19