66

I work in engineering and I was warned by a colleague years ago that I should not cite materials on arXiv because they are not peer reviewed. Although it was never clear to me what peer reviewed meant because the papers on there seemed to be well maintained and the ones I have came across has been theoretically sound. Academia stackexchange certainly didn't help in relieving some of those anxieties with people asking if they are "ruined" for posting on arXiv.

One can understand my point of view better by considering that in engineering there are well established conferences and list of papers published through those conferences (although not all high quality). Browsing through a list of citations published by engineers, there is a definitive lack of "arXiv" related papers. This is exacerbated by the fact that arXiv papers seem to be less applied and more theoretical, "mathy", so they are largely hidden from view for the typical engineer.

But from my recent experiences, it seems that a few fairly notable researchers in my field have extensive track record of publishing materials on arXiv. People such as Steven Strogatz who works in nonlinear dynamical systems and John Baez. I have hypothesized in the past that very well established researchers are submitting papers on arXiv instead of science journals because they would like a wider audience, and undergraduate researchers are submitting on arXiv because there is no other venue for them. But of course I am not sure if that is indeed the case.

Now I am currently reading some material published on arXiv by someone who I am not very familiar with. I am still not 100% confident on the "quality" of the paper i.e. scientific accuracy of theoretical and experimental results. But I believe that I would be more open to using materials on arXiv in the future if I can understand the motivation behind submitting to arXiv i.e. whether if they are just low quality papers without any other place that accepts them.

Can someone please offer a break down of some the reasons why one would publish to arXiv?

Fraïssé
  • 11,452
  • 14
  • 62
  • 97
  • Another aspect considered is that science is going into open-access, and arxiv encourages open acces movement in science. – Nikey Mike Aug 17 '16 at 10:40
  • 5
    Your colleague is wrong about not citing papers posted to the arxiv because they are not peer reviewed. You should cite whatever you use and trust, whether it has been peer reviewed or not. If you can't judge for yourself whether a source is reliable, then probably you shouldn't be publishing. – Dan Fox Oct 20 '16 at 07:21
  • @DanFox doesn't what you said apply to Wikipedia as well? – Ooker Mar 13 '18 at 06:43
  • @Ooker Sorry I'm a little late on this. But Wikipedia is to a research paper is what a textbook is to an experimental journal. One could say Wikipedia articles are a compilation of scientific knowledge. Now were they come from, research, is documented in a paper. So citing a paper (peer-reviewed or otherwise) isn't exactly the same as citing Wikipedia articles. –  Jul 17 '21 at 19:21
  • That said however, there is a difference between citing arXiv papers vs peer-reviewed ones. @DanFox , remember, researchers get it wrong too. They only publish THEIR thoughts, theories and findings on things. So the more you have people criticizing a paper, the more likely you are to find fallacies, paradoxes and such (essentially the idea behind peer-reviewing). While citing arXiv papers is nothing to be discouraged (as long as you trust the information on the paper), you are definitely more likely to have cited "bad" (for a lack of better term) papers. –  Jul 17 '21 at 19:30

3 Answers3

75

The major use case of arXiv is for disseminating manuscripts that you also publish in a journal or conference. By posting a preprint on arXiv, people can find your research, build on it, cite it, and give you feedback on it immediately, while at the same time the same work goes through the (sometimes slow) peer review process. Some of these papers will fall out of the peer review pipeline at some point, and only appear on arXiv, but that doesn't necessarily mean that they are less useful, important, or sound.

To give a specific example, in the last year or so I have been working in an area so new that most of the relevant research is still only available on arXiv. I expect these papers will eventually appear in journals too, but the slow peer review process means that the latest journal issues do not represent the state of the art for this particular topic. Conferences have a quicker peer review cycle and are more current, but most only publish relatively short papers.

arXiv is also useful for work that is in a format not suited for a conference or journal (e.g. a thesis), or for extended versions of papers that are published somewhere else.

I would caution against rules like "I should not cite materials on arXiv because they are not peer reviewed." Peer review does not guarantee sound, high quality research, nor is the inverse true. You should critically evaluate each paper, peer reviewed or not, on its individual merits. (Also see this related question on Math Overflow.)

ff524
  • 108,934
  • 49
  • 421
  • 474
  • Good eye-opener for the use of arXiv; +1 for the last paragraph. Coming back to arXiv papers getting published as journals, would there be a way of aliasing to combine the citations of the two versions? Also can anyone just put their thesis in arXiv (for the visibility), or do you need your institution permission to do so? – Ébe Isaac Aug 17 '16 at 09:37
  • 2
    @ÉbeIsaac As far as I know, Scopus does not count citations to the Arxiv version of a paper. Which is one reason to cite the published version (if it exists) rather than the Arxiv one. – Federico Poloni Aug 17 '16 at 09:42
  • @FedericoPoloni: how about Google Scholar? – Ébe Isaac Aug 17 '16 at 09:51
  • 1
    "You should critically evaluate each paper, peer reviewed or not, on its individual merits." If don't use peer review as a guideline on what's worth your critical evaluation, you basically deny the sense of peer reviews. Also, your life would end before you'd manage to find anything useful. – BartoszKP Aug 17 '16 at 09:51
  • @ÉbeIsaac Google Scholar counts citations to both, usually. But that's irrelevant to my point: the fact that one common scholar database requires citations to the published version is enough reason to prefer them, in my view. "We hired X rather than Y also because he has more citations on Scopus" happens, unfortunately. – Federico Poloni Aug 17 '16 at 09:56
  • 2
    @BartoszKP: I disagree. Peer reviews are useful, but not mandatory. Many peer-reviewed publications have serious flaws slip through them, even in reputable journals. At cases, even a gist/idea can help look at a problem at a new angle to solve it, it does not necessarily require a slow peer reviewer to valuate that. Sometimes (actually a lot of times), the process can even take up to 4-6 years. In such cases many methods tend to get outdated even before they are published. – Ébe Isaac Aug 17 '16 at 10:07
  • 1
    @ÉbeIsaac I didn't say they are mandatory - I meant they are useful, because I understood the part I quoted as a suggestion that they are not useful at all. Of course it may be beneficial to browse&skim through non-reviewed papers, but definitely not "critically evaluate" all of them (well, depending on your field of course and the number of papers available). – BartoszKP Aug 17 '16 at 10:17
  • 2
    I would upvote this question so many times, if I could, even only for the "Peer review does not guarantee sound, high quality research, nor is the inverse true.". Finally someone who understands that peer review is not the truth come down to Earth. – gented Aug 17 '16 at 13:39
  • @BartoszKP Personally, I don't find "has been peer reviewed" to be a useful heuristic in deciding what papers to pay attention to and "critically evaluate". Of course I won't spend time on all non-reviewed papers; I also won't spend time on all reviewed papers either. Instead, I decide whether to spend time on a paper based on its apparent relevance, and whether on first (quick) glance it seems well written and sound. – ff524 Aug 17 '16 at 13:47
  • @ff524 "Personally, I don't find "has been peer reviewed" to be a useful heuristic". You are free to do it, but again - this means your denying the sense of how science currently works. Also, in many universities (don't know about yours, of course), papers that weren't peer reviewed are not allowed, or at least are strongly discouraged as thesis references. – BartoszKP Aug 17 '16 at 14:00
  • As to whether you need institutional permission to submit a thesis to arXIv: depends on the institution, its policies, and local law. Some institutions assert copyright or other control over their students' theses; in that case you would need their permission to publish or distribute the thesis or its contents anywhere. At many other institutions the student retains full ownership of the thesis, and is free to submit it to arXiv or anywhere else they want. – Nate Eldredge Aug 17 '16 at 15:20
  • @BartoszKP "You are free to do it, but again - this means your denying the sense of how science currently works." do not mistake the mere act of publishing tons of material with actual science: they are two totally different concepts. – gented Aug 17 '16 at 19:43
  • @GennaroTedesco I don't understand what do you mean. Publishing tons of material that wasn't peer reviewed is exactly the thing that I presented as not a good example of how science should work. I wasn't even referring to what science is, just to how the process currently works. And it works thanks to peer reviews. Of course you can argue about it not being ideal (i.e. publishing tons of peer reviewed material, but somehow not very good quality), but still no better&practical&implementable solution have been found yet. – BartoszKP Aug 18 '16 at 10:08
  • @BartoszKP I was referreting to the converse of the statement (the one that you highlighted in brackets), in particular the process of publishing material that has been indeed peer reviewed is different from doing science. I wouldn't take the peer reviews as assessment of what is or is not scientific. There are many examples of strongly good research (even outstanding results) that hasn't even been submitted anywhere (by choice of the authors) and billions of example of high school quality material that for reasons still unkown makes it to journals and similars. – gented Aug 18 '16 at 10:15
  • @GennaroTedesco I didn't say that publishing is science. I'm only saying that peer reviews are, in general, the only sensible heuristic we have currently (see my remark that obviously it's not ideal in my previous comment). In some specific fields it may be possible to evaluate everything that is published yourself, but in general it is not possible. And also - see my previous remark on how this is handled formally on universities. I repeat: I agree that it's not ideal, but you cannot just write that it's basically worthless and you shouldn't rely on it at all. – BartoszKP Aug 18 '16 at 10:52
  • @BartoszKP I use wikipedia everyday and it is tremendously useful. How did you come to the believe that peer review is the only way to create useful content? The internet is full of useful content, and no -- I don't spend my whole life looking for content: with google search it takes seconds to get LOTS of useful material, which is never peer reviewed. On the other hand, I have seen lots of BS in peer-reviewed journals... – logical x 2 Sep 07 '17 at 10:53
  • @deusexmachina Wikipedia has a form of peer review and quite strict policy on which sources you should cite (guess what types of publications are deemed as best sources). You were very keen to mention that a) useful content without peer review exists b) some peer reviewed content is low quality, but somehow you've failed to account for the fact that c) useless, low quality content without peer review exists also. So, to be fair we would have to compare not only the amounts of useful content created with PR and without, but also the amount of garbage you need to dig through to find it. – BartoszKP Sep 11 '17 at 11:26
  • @BartoszKP Well, wikipedia has a form of peer-review, but it is not exactly the classical journal-type of peer-review. In that sense, arxiv also has some form of "peer-review", as offensive material will be removed etc. The point is, that projects like wikipedia (and SO, by the way) show that an alternative to the classical peer review model (e.g. everyone can comment/upvote) does provide useful content. – logical x 2 Sep 11 '17 at 12:03
  • @BartoszKP For example, here on SO, I don't have to dig for my whole life to find useful answers ; ) In fact, I find useful material much quicker on the inet as compared to papers. Turns out, the only way to get through the vast amount of scientific literature are web services like google scholar or scifinder. So, I really dont get the point that peer review helps in finding content quickly because that just isn't the case. Ever read through all Nature articles of the current month? The only question is if we want a transparent model for exploration or proprietary services like google scholar – logical x 2 Sep 11 '17 at 12:07
  • @deusexmachina Not in a scientific context. I agree that the examples you mention provide useful and valuable content, but this is a bit different level. And the fact that both of these examples do have a form of something like peer review, strongly suggest that this is the key. Mentioning on every occasion that PR content that is low quality exists is simply childish - there are also worthless Wikipedia articles and worthless SO posts with high score. Of course that no perfect process exists. But the more some content is verified and cross-examined the higher chance is that it's accurate. – BartoszKP Sep 11 '17 at 12:08
  • @deusexmachina I didn't mean searchability in the context of matching your problem at hand, but in the sense of quality. When I'm reading Artificial Intelligence from Elsevier I'm confident that each of the presented papers is high quality work, provided by experts and verified (and yes, accidents happen and there are some flaws in this process). This is only sometimes the case for SO, when most of the time you have answers by "random" people, of which you know only that only probably where helpful to the OP. Nothing wrong with it - SO serves a bit different purpose than scientific journals. – BartoszKP Sep 11 '17 at 12:14
  • @BartoszKP Quality is a matter of context. I just don't agree with the view: "Also, your life would end before you'd manage to find anything useful" Basically, you depend on non-peer-reviewed material all the time, and it is indeed quite useful ; ) – logical x 2 Sep 11 '17 at 12:25
  • @deusexmachina "you depend on non-peer-reviewed material all the time". Not if you want to do science on a professional level. – BartoszKP Sep 11 '17 at 15:52
  • @BartoszKP No? You never use google? Never? Do you believe the earth is flat? How do you know for sure that it isn't: Did you look it up in a peer-reviewed journal? At the end we all trust the majority vote: the majority vote becomes the truth -- and I rather have the truth of 2-3 billion people than the truth of 2-3 reviewers ; ) – logical x 2 Sep 11 '17 at 19:58
  • @deusexmachina My initial comment was referring to the concern of "citing" not, "looking at". Also, I've explicitly mentioned that it's a obligatory guideline, not rule to follow. "and I rather have the truth of 2-3 billion people than the truth of 2-3 reviewers ; )" - even if most people suck at math and don't have a clue about medicine and these 2-3 reviewers happen to be experts in the field? Good luck. – BartoszKP Sep 11 '17 at 20:33
12

Part of your question seems to be "Why do some researchers submit to arXiv while others don't"?

A point not yet mentioned: arXiv only covers a few specific subject areas. It's the very first line on the home page:

Open access to 1,175,314 e-prints in Physics, Mathematics, Computer Science, Quantitative Biology, Quantitative Finance and Statistics

Since engineering isn't on that list, that would be an excellent reason for most engineering researchers not to post their work to arXiv.

The "notable researchers in your field" who you mention as being prolific arXiv posters are, in some sense, not really in your field. Strogatz is a mathematician and Baez is a mathematical physicist. Both those fields are covered by arXiv, so it makes sense that they would use it.

Now it's entirely possible that although you think of your field as engineering, some of your papers might contain "enough" physics or mathematics to be on-topic for the corresponding sections of arXiv. But you might have to think about it first.

It should also be strongly emphasized that these people are not submitting to arXiv instead of peer-reviewed journals, but in addition to. This is perfectly acceptable by the standards of most journals in mathematics and physics. There are plenty of peer-reviewed outlets for their work and they're using them. Just look at their CVs.

Nate Eldredge
  • 133,015
  • 44
  • 379
  • 480
  • 4
    Now it's entirely possible that although you think of your field as engineering, some of your papers might contain "enough" physics or mathematics to be on-topic ... — In case folks are wondering, in electrical engineering, for example, researchers working in information theory, machine / statistical learning, control, and on various optimization problems post quite a bit on arXiv. – Mad Jack Aug 17 '16 at 15:16
7

The most important thing is to know if peer-reviewed journals in your field will accept to publish your article if it has previously been published on arXiv.

In my field, chemistry, the answer is no, so publishing on arXiv has never been an option. Note though that some publishers allow an accepted paper to be published on arXiv, but that is after it has been accepted.

SteffX
  • 196
  • 5