4

This question is not about the politics of scientometrics, nor is it on the role of scientometrics in governmental research programs, nor is it about the use of scientometric indices: these questions have been already addressed and debated somewhere else on academia.SE., for instance

The following question is about scientific method(s).

After these few words to avoid any debate, I would like to naively ask about the scientific rigor of scientometrics. Is there any kind of rigor/general method/scientific concept behind this field of research? Or maybe a more aggressive question: is scientometry a science?

Community
  • 1
FraSchelle
  • 321
  • 1
  • 7

1 Answers1

14

Scientometrics is a science to the extent that it applies the scientific method to a field of inquiry. Researchers in this area formulate questions and conceptualize existing problems (e.g., “in these times of scarcity, the public wants research to be efficient: how can we measure this?”), they make hypotheses, make predictions, gather data to test them, analyze the data and prove or disprove their hypotheses.

Regarding “what do we scientifically learn from scientometrics?”, well, there are plenty of established results (you can find some on wikipedia), but I'll use one recent paper from Scientometrics which highlights what this field can bring us:

Negative results are disappearing from most disciplines and countries
D. Fanelli, Scientometrics 2012, 90, 891–904

This one was really an eye-opener for me, on a phenomenon which I always supposed existed, but it was nice to see it backed by hard data. Other examples include:

Physical and economic bias in climate change research: a scientometric study of IPCC Third Assessment Report
A. Bjurström and M. Polk, Climatic Change 2011, 108, 1–22

 

Language biases in the coverage of the Science Citation Index and its consequences for international comparisons of national research performance
T. N. van Leeuwen, H. F. Moed, R. J. W. Tijssen, M. S. Visser, A. F. J. van Raan, Scientometrics 2001, 51, 335–346

I'll finish with a personal opinion: while I am annoyed, as most people, with the emphasis currently given on bibliometrics in the evaluation of research and research projects, I think scientometrics in general has an important role to play, just like related fields such as studies on ethics of research: better understanding the positive and negative implications of the way we currently do science is healthy (the meta-level of research).

F'x
  • 73,069
  • 23
  • 262
  • 392
  • Thanks a lot for answering this naive question. Thanks also for giving so good examples of applications of scientometrics. Thanks finally for your last remark regarding the difference between bibliometrics and scientometrics. I nevertheless wonder if it exactly answers my question. Your three examples are for sure much more interesting use of the statistical methods than the simple, almost stupid counting of papers I usually heard about in bilbiometrics (h-index, impact factor, ...). Did you really mention that one can make predictions ? Could you please tell me more about that ? – FraSchelle Jul 26 '13 at 15:19
  • (cont.) If I try to compare scientometrics (with the few knowledge I have about it) with an other (naive) topic where I hope to find statistic and social studies both together, say economics (what a big word, isn't it ?), I have the feeling scientometrics still misses the point, there is no description there, in the scientific sense. Let me put an other naive picture: if I want to compare scientometrics and weather forecast, I would say that they are almost identical, except scientometrics has nothing like the fluid equations at its disposal. My question is: am I missing the point or not ? – FraSchelle Jul 26 '13 at 15:26
  • @Oaoa have you read the PNAS paper defining the H-index? – StrongBad Jul 26 '13 at 15:51
  • @DanielE.Shub Thanks for the link. This one http://arxiv.org/abs/physics/0508025 is for free. Well, when I discuss this paper with colleagues (I'm physicist), we usually do not understand why people uses the h-factor and not the m-one, as discussed by Hirsch. – FraSchelle Jul 26 '13 at 16:36
  • 1
    @F'x +1 for such a nice answer. – Shion Jul 26 '13 at 18:48
  • 1
    @Oaoa There is a difference between proposal of a metric or any kind of a measurement system and the adoption of one. Adoption of a metric is socio-political in nature and rather subjective. – Shion Jul 26 '13 at 18:49
  • @Oaoa Your previous comment betrays an almost contemptuous disdain of mere "counting" as you put it. Why is counting "lower" in the scientific method than "predictions?" The scientific method makes no such distinctions and thus, your statement would be an opinion against the scientific method and not a fact towards it. – Shion Jul 26 '13 at 23:48
  • @Shion I then urge you to understand what science is about. Sure counting and predictions are at the same level, in the sense that when counting predicts nothing, it is instantaneously abandoned. Usually it means you have to change your investigation method(s). When a prediction is not verified by counting, it has to be abandoned as well. Of course, I merely replaced here counting=experiment and prediction=theory, which is a kind of abuse, since nobody here can tell me to what extend scientometric is a science, whereas the comparison seems to follow this line. – FraSchelle Jul 28 '13 at 08:50
  • Regarding the naive-counting (h-factor, impact-factor, ...), of course I hate it ! As the nice F'x's answer shows us, there are clearly more interesting results when you need nice methods of extracting the relevant infos from a mass of data. The usual counting factors can be computed in mass by any computer, they require almost no effort, and they mean nothing. At least, it's the same for me as saying a scientific publication is a piece of paper. – FraSchelle Jul 28 '13 at 08:54
  • (cont.) I believe they are much more than a single piece of paper that you pile up somewhere, and I would like here to understand how people working in scientometrics see them at the moment. I also believe scientometrics is not only about papers, but also on the long, laborious intellectual process(es) done before publishing. The first example of F'x (about the disappearance of negative result) clearly goes along this line, since it clearly say that, before publishing, researchers exclude they negative results. That's an interesting way of saying this result I believe. Thanks for your comment. – FraSchelle Jul 28 '13 at 08:57
  • 2
    when counting predicts nothing, it is instantaneously abandoned. — False. Not all science is predictive, or even claims to be. Also: some true Scotsmen despise haggis. – JeffE Jul 28 '13 at 09:16
  • @JeffE Well, I think I'll make no more comment, since it's totally despairing. Note nevertheless, the first sentence of Wikipedia [ http://en.wikipedia.org/wiki/Science ], Science entry: Science (from Latin scientia, meaning "knowledge") is a systematic enterprise that builds and organizes knowledge in the form of testable explanations and predictions about the universe. – FraSchelle Jul 28 '13 at 16:10
  • 1
    @Oaoa I won't go into a didactic argument with you on the definition of science since JeffE gets at its heart in his previous comment. The wikipedia definition of science you link to is limited. It ignores inductive, data driven compilation of knowledge as opposed to the traditional, deductive, hypothesis driven approach which that definition alludes to. – Shion Jul 28 '13 at 17:20
  • I just want to comment that bibliometrics isn't merely "evaluation of research and research projects" (not sure if this is what you meant though), but more fundamentally it is analysis (statistical, systematic) of publications. As such, it is (sort of) a subfield of Scientometrics (or a method used in scientometrics). It often uses citations, citation relationships, or other bibliometric connections, but it can also overlap with linguistics, information retrieval and other fields. – fileunderwater Oct 18 '16 at 09:10