35

EDIT Since there are some misunderstandings regarding my question I'll try to rephrase, and also avoid posting questions late at night when I am really tired :)

A typical part of my work as a bioinformatician/data analyst is to check the relevance of my results, when I am drafting my articles, by doing a series of literature searches to see how my results relate to the field in general. This typically takes me beyond my own competencies, and into fields like histology, oncology, tumor biology etc..

Quite often I run into very interesting results presented in articles that were published in obscure journals with pretty low IF (say for example 1.0-1.5). Sometimes the authors turn out to be from a rather unknown university from a unexpected country. I realize that this is a bit controversial, and I really don't mean to be looking down on anyone's creativity or work ethics but biomedical research is usually expensive, with all the lab consumables, customised reagents/antibodies/proteins/peptides etc..

One such example was an article I found yesterday where the authors claim prognostic potential for a particular protein which also happens to be significantly regulated in my dataset. Naturally I was excited at first but then two questions arose:

  1. Could these people have really done what they claim they have done?
  2. If the work is legit, then why it ended up going to a journal that's so obscure, considering that the findings might be very relevant for patient care. Cancer and biomarkers are two "hot" fields and there are literally lots of well known avenues for publication, before you come to think of this one, or this one. If the results are as interesting as I think they are then they should have surely been published in a venue where they'd get more attention.

Am I being too harsh to be suspicious?

posdef
  • 17,711
  • 5
  • 75
  • 125
  • 4
    Regarding unexpected places: I know that Pakistan is making an effort to boost research by forming PhD students abroad and bringing them as faculty. I personally know a bunch of bioinformaticians from there in Stockholm that will bring back a well connected research network. – Davidmh Jul 06 '15 at 03:46
  • 3
    I am not from the biomedical field, but are any plausibility checks, other than re-performing everything, really impossible? I don't think they would be possible, otherwise how would journals publish anything at all? They don't sit down and reproduce everything you say, and neither are they simply taking your word for it! – 299792458 Jul 06 '15 at 04:26
  • 7
    Take into account that for some fields an IF of 1.5 is not quite low: the IF depends also on how many people works in that specific field. For example in my field all the top journals have IFs between 1.5 and 2. – Massimo Ortolano Jul 06 '15 at 08:42
  • 2
    @MassimoOrtolano That is definitely true.. Good call to point it out! however in this case, 1.5 is definitely subpar. Nothing I have done so far could be considered high impact and I'll be the first one to admit it, but I'm typically going after something at the 4-5 range for a typical study. – posdef Jul 06 '15 at 08:54
  • 31
    It is not only normal but necessary to be suspicious of every really interesting finding, no matter where it is published or by whom. – JeffE Jul 06 '15 at 14:38
  • 8
    Let's take an example from the other end: "vaccines cause autism" was published in The Lancet with IF 45. It's never good to focus too much on the journal reputation but not enough on the research itself. – Agent_L Jul 07 '15 at 11:42
  • Are you suggesting that Spain and Japan don't have money for scientific research? That these are somehow "backwards nations"? WTF? – Lightness Races in Orbit Jul 08 '15 at 11:30
  • @LightnessRacesinOrbit That was not at all what I was suggesting, the two links you are referring to are two examples of the low-IF venues I was talking about. – posdef Jul 08 '15 at 11:35
  • Yes, I got that... – Lightness Races in Orbit Jul 08 '15 at 11:35
  • 1
    @LightnessRacesinOrbit in that case your comment above is rather unnecessary, don't you think? – posdef Jul 08 '15 at 12:06
  • @posdef: No. Your odd and offensive ethnocentric views are. – Lightness Races in Orbit Jul 08 '15 at 12:10
  • 1
    @LightnessRacesinOrbit you either misunderstand what I wrote, or choose to interpret it in an odd and offensive way. Since you seem to show no interest in figuring out what I was trying to say or my motivation with the question, despite my efforts in trying to explain my thoughts, I say it doesn't matter either way. Have a nice day :) – posdef Jul 08 '15 at 12:40

8 Answers8

44

I am concerned by your choice of words:

  • "Obscure": one person's obscure may be another person's core community.
  • "Low IF": IF is very dependent on both community and "trendiness." A lot of very good work is published in low IF venues because they are the appropriate place to publish the result.
  • "Unknown ... and unexpected": again, not unusual at all for something interesting to come out of an unexpected place.

All of these words, to me, point to lazy "popularity-contest" thinking in evaluating the context of a work. Instead, I think that you should use more scientifically meaningful heuristics, such as:

  • Does the investigator have a history of solid work, into which this result fits?
  • Does the venue have a clear, well-defined focus and community, and does it have reputable management?
  • Is the work well-suited for the venue?

If a work doesn't pass these filters, then I think you should be highly suspicious and not bother digging more deeply. If it does, however, then you should be just as skeptical as you should be for work coming out of a high-prestige laboratory in a leading institution---and no more.

jakebeal
  • 187,714
  • 41
  • 655
  • 920
  • 7
    Yeah, I agree entirely with this answer. I had been reading the OP as talking about "shady journals" or ones that, from his knowledge of the area, he knows are not the sort that breakthrough results get published in. Just going by the impact factor: that's for administrators who can't possibly acquire a clue in the needed amount of time. Be a professional and acquire a professional opinion. – Pete L. Clark Jul 06 '15 at 02:40
  • 3
    I fully understand your concern, as I said I'm aware that it might give the wrong impression. Anyways I tried to clarify some of the misunderstanding with some additional text to the question body. RE: your answer - the issue stems from the fact that I am beyond my own competency here, in other words I cannot judge how solid the immunohistochemistry is, or if there's anything fishy with their choice of antibodies. Looking at the source and venue of the publication, I became suspicious since I cannot easily say if they pass the test, with what your call "heuristics" – posdef Jul 06 '15 at 08:10
  • 1
    The impact factor, in biology, is not a bad proxy for journals' acceptance standard. – Cape Code Jul 06 '15 at 16:53
  • 3
    @CapeCode Two major problems with that: 1) "Low IF" is in the eye of the beholder. I know people who snoot at a 5.0 IF because they're always gunning for Nature/Science/Cell. 2) Audience size is not the same as acceptance standard. Consider, for example, the SJR rankings for ecology journals: a lot of quite good journals have low IF simply because they are specialized. – jakebeal Jul 06 '15 at 18:17
  • @jakebeal wrt specialized - according to the OP, the results could be very relevant to patient care. They should be broadly communicated to everyone interested in patient care and not restricted to a highly specialized audience. – emory Jul 06 '15 at 21:22
  • @emory We can't know without actually reading the paper ourselves---see the nice answer by anon. – jakebeal Jul 08 '15 at 10:14
31

I don't think it's too harsh to be suspicious. I think that suspicion should be the default state of academics appraising each other's work. Nor do I think that you need to be equally suspicious of everyone: if certain indicators make you more suspicious, go for it. Having "a nose for the truth" is part of being a successful academic, and I don't think it's worth apologizing for.

The question is what to do with your suspicion. In my field -- mathematics -- if you get suspicious of something, the remedy is in principle easy: you read it and check it. In reality this can take a lot of time and energy, but the more important the result is the more necessary and valuable that time spent will be. Note also that it may have the effect that you come out having completely understood the work, and then you can vouch for their work, so your suspicion will be a net positive for the authors. (And if their work turns out to be flawed and invalid, they should still thank you for your efforts. Even if they don't, your conscience is clear.)

I'm afraid I don't really know what you should do in a biomedical field for which reproducing the projects would be an undertaking involving more capital than just your own time. I would guess that a sufficiently important biomedical study would get rechecked and refined by others. But if you really just think "I highly doubt that if I did that study correctly, I would get the result that I read about, and if I didn't, then it will have been a waste of time" then depending upon how skeptical you are it seems reasonable to contact the authors and ask for more information or simply ignore it altogether. The last sounds harsh, but it seems to to be necessary: what else are you supposed to do, go slavishly down a path that you firmly believe is wrong?

Added: @Kimball asks whether by "ignoring" these papers I mean not to cite them. Good question. In my opinion, just because paper A is published in field X and you are writing a paper in field X does not force you to cite A. If your work contradicts A then you still are not necessarily required to cite it: e.g. when you write a paper about the transcendence of Pi you will probably not cite papers purporting to show that Pi is algebraic or rational. You need to cite paper A if you use or rely on it in any way. You probably should cite a paper A if you think that others in your field will be aware of it, take it seriously and that a lack of citation would create confusion in the field. I would say that except in the precise circumstances enumerated above, you should not cite work that you believe but do not know to be false.

Further Added: I think that @The Dark Side made a crucial point in a comment above. In every technical field, papers get refereed, and serious/breakthrough papers get seriously refereed. This process however does not involve nontrivial expenditure of capital or systematic, large-scale reproduction on the part of the referee. But the referee must do something, and I would hope that the authors' institutional affiliation plays a small role in the evaluation process (ideally: no role at all). So the generalization of the answer above from mathematics to other academic fields seems to be: if you're suspicious of a potentially paper, you should re-referee it. In this particular case the OP seems to lack the expertise to fully do that. Okay, that's what friends and colleagues are for.

Pete L. Clark
  • 130,631
  • 25
  • 361
  • 522
  • 3
    Besides ignoring it completely or reproducing the experiment, there's also the option of saying "that's interesting, if so" and waiting for someone else to make the effort to reproduce the results. Which is the same thing we do -- or should do -- with claims outside our own field of expertise. – keshlam Jul 06 '15 at 00:39
  • 1
    I think the OP may be concerned about how/if to cite some of these works in their own paper. Are you saying to simply ignore it in this context also, or just that you shouldn't rely on it a basis for your research? – Kimball Jul 06 '15 at 01:11
  • @keshlam: I agree. I only advocated "ignoring completely" results that you feel couldn't possibly be right. Whether there are any such results in the OP's field, I am not sure. But for instance about once a week a paper gets uploaded to the math part of the arxiv for which the only reasonable reaction is to completely ignore it: such papers are surely false (in some cases they directly contradict other work) but nailing down their falsity is more trouble than it's worth (e.g. because the exposition is so poor). Unfortunately many such papers are now being published in "obscure journals"... – Pete L. Clark Jul 06 '15 at 01:34
  • An obscure result in an obscure publication probably deserves its obscurity. There are certainly exceptions, but ... – keshlam Jul 06 '15 at 01:34
  • A while back, I found a very "nice" paper about integral transform theory in a very low impact journal. It's work that is tangentially related to mine. I've read it several times and I'm still not very convinced about the material therein. Both authors respected mathematicians but they cut a lot of corners in the paper (stated things without proof). I'm even fairly suspicious of the journal. It hasn't made the list for predatory journals but it doesn't seem to be very rigorous. Stuff like this happens sometimes unfortunately. – Cameron Williams Jul 06 '15 at 15:39
  • 1
    @Cameron: In truth I think there's a continuum here. I know of at least one journal that publishes a lot of papers of the following kind: the results they contain are exciting breakthroughs in the field; the exposition is not careful enough to make it easy to tell whether the proofs are correct; in fact the proofs are manifestly wrong or incomplete in some superficial ways; nevertheless the paper, flaws and all, is clearly of value. Caveat lector, to be sure. – Pete L. Clark Jul 06 '15 at 16:08
12

concerning your remark:

Quite often I run into very interesting results presented in articles that were published in obscure journals with pretty low IF (say for example 1.0-1.5). Sometimes the authors turn out to be from a rather unknown university from a unexpected country.

I feel that it may be difficult for 'unexpected countries' or 'rather unknown universities' or 'underexposed research groups' to actually publish in a highly ranking journal just due to this suspicion, which also bugs the reviewers and journal staff.

Also it happens in reverse. Well known groups can sometimes just a little bit too easily publish a sloppy experiment in a high ranking journal.

In that sense, I think this suspicion is overrepresented in literature, and we should be careful to apply it too much...

Maarten Klop
  • 121
  • 4
  • 1
    It's a good point definitely, it's a vicious cycle if the lesser known groups are discriminated against and all we have left is a "rich get richer" scheme.. I feel it would have been so much easier if there an easily accessible *track record* for a publication, where one could see if a the work has bounced from several highly reputable journals and settled on a less known journal, or if it directly went into that journal because the authors feel it's the "right forum" for that paper. – posdef Jul 06 '15 at 08:58
  • This danger probably exists. But since academia works like a guild, I feel it's truly hard to learn to write good papers / do good science, unless you learn from a successful scientist. (Affiliation is a mediocre proxy for this, since scientists move).

    Last time I heard such a complaint (from a researcher of little fame who had an actual cool finding), the complaining researcher added "reviewers made me waste tons of time because they forced me to make my paper self-contained". With such ignorance of how to write a paper, no wonder publishing it took so long.

     – Blaisorblade Jul 07 '15 at 20:44
5

This typically takes me beyond my own competencies, and into fields like histology, oncology, tumor biology etc..

When I want to evaluate the plausibility of a result from outside my area of expertise, I typically look for a friend or colleague who has the necessary expertise, or who might be able to point me to someone who does.

I think this is much more likely to yield meaningful results, and much better for science as a whole, than evaluating an article I'm not competent to read based on the impact factor of its journal or the authors' institutions and countries of origin.

To put it more harshly: if you're not competent to do something, don't do it. Instead, collaborate with someone who has the skills you need.

Vectornaut
  • 629
  • 4
  • 8
  • 1
    I would normally agree on the notion in your last sentence, and that is how I normally approach this problem, however based on how often this situation occurs, it's practically not feasible to ask to a colleague every time I run into something I cannot evaluate. It might be how things are done at my lab... – posdef Jul 07 '15 at 00:05
5

I think that the results are simply not as interesting as you think they are. A result from a new (while perhaps theoretically sound) statistical technique or data set that has not been subjected to rigorous cleaning (because if it were then there would be nothing left of it).

As far as having the equipment goes, I'm not sure exactly what papers you're looking at, but from the standpoint of someone doing some biostatistics research at the moment, we often make use of data that has been made public or that which is obtained through collaborators as we have no wet lab. We have to wade through cleaning that data properly (which is extra difficult because it is hard to gather reliable biological data in a good form even when the biologist knows what is needed, many do not). We are often trying very new techniques, between potentially bad data and techniques that have not yet gone through the proving grounds... no matter how rigorous we are we will have some doubts about the result.

If we are the first with such a result we would not be willing to trumpet it to too high IF journals. The results are not interesting to us unless:

  1. it also reproduces good findings from others besides the new thing it finds or
  2. the results are experimentally confirmed (which would usually be quite a while after publication, though alternatively a more biology based collaborator with a wet lab may do so).

Your work is the reason their work is interesting, it's not interesting on its own. Balance this with some healthy skepticism in general of course though, statistics has a special case of irreproducible results, unreasonably so.

Unlike some other posters, I believe that the magnitude of their results may not be very apparent at all to them in the case where they actually are major. This is because, like yourself, they may not have a deep knowledge of the field that the math is being applied to, they run their code on some data they got and oh, it turns out gene xyzl_at is significant, that holds no meaning for them in the way it would for a biologist working on the disease.

posdef
  • 17,711
  • 5
  • 75
  • 125
anon
  • 51
  • 1
2

I think you are right to be suspicious - but as long as the journal is included in those abstracted by pubmed etc then I you should still include it in your review. Just put a "flag" by it (if you really don't like it put it in an appendix). Including the paper shows; a) your search was extensive b) you have some criteria for judging value (you've flagged it)

There are "social" reasons why a good paper might appear in a low IF journal, eg: the author needed a publication quickly 'cause they were going for promotion, the author had a "spat" with one of the "silver backs" in the field, the author lacks confidence in their own ability or has been undermined by "colleagues", the institution they work for might be "behind the curve" in the obsession with IF etc etc.

Richard Wadsworth
  • 21
  • 2
1

We all know that IF is a very problematic way to judge work quality, and most of what happens in science is not published in the "glamour" journals. We can also agree that every paper should be judged by its content and not by the journal it is published in. Finally, the notion of importance can be very subjective.

However:

I completely understand your concern and I think it is legitimate. I don't ever recall meeting a biologist that had an important result and would want to have it published in an extremely obscure journal. And yes, in biology even within specific subfields there will be several journals with IF higher than 1.0-1.5 (in this case the IF is not an indicator of quality but often is an indicator of what other people read). To be even more blunt, I don't remember ever seeing important results in a journal with this kind of IF. To me, this suggests that if the work ended up in such a journal, it is very likely the authors couldn't get it accepted in a "better" journal, so at least some people thought there was something wrong with it.

The bottom line:

It is technically possible that the results are very important and the author didn't care where they were published or the importance of the results was not realized. However, this is extremely unlikely in my experience.

Bitwise
  • 10,813
  • 1
  • 41
  • 64
1

The only two valid metrics you should use to assess individual papers, as a rule of thumb, are recency and number of citations. When it comes to recency, the results might become disputed in the future, but it is possible that the paper reflects the latest developments. When it comes to number of citations, this is often a clear indicator of the impact of the paper. Web of Knowledge or Google scholar can provide you with a citation metric. For rapidly moving fields, such as the natural sciences, 3 years is an acceptable cutoff for recency, but may vary from field to field. Generally, whether a paper is either "classical" or "fresh" is used as a criterion for inclusion as a potentially citeable source (e.g. for a literature review) and not for assessment of quality.

Due to the change toward open access the IF and related metrics are starting to become less valid. In addition, they are controversial. They can be useful for authors planning to publish, but without the background knowledge required to evaluate the validity of the paper, what others think becomes more important. Thus, the measure of the individual impact of a paper should be consulted. You could potentially use IF indirectly, by checking what the impact factor of the journals citing the paper amount to. However, citation does not imply that the paper is a "good" paper. For example, the paper by Stanley Milgram "Behavioural Study of Obedience" has been cited 3847 times according to Google Scholar, but it is a text book example of when research ethics goes wrong.

noumenal
  • 321
  • 1
  • 5