4

I was using Google Scholar to judge the rank of a researcher, but then realized I couldn't find everyone I wanted to and came across this question which highlighted this problem. Looking at Scorpus, which was suggested in that same answer, seems to downvalue any non-Elsevisor publications and not have any current data (from the last year). Microsoft Research seems altogether unintuitive to use and doesn't seem to have h-index at all. I tried PublishorPerish, but filtering data to find the correct researcher's papers seems ridiculously difficult.

I would like to be able to judge status somewhat programmatically, understanding there may be some flaws in this method, as it would be somewhat cumbersome to do it by hand on the scale I would like to. I will probably supplement it my sorting the top 3-5 by hand, but I would not like to do the entire set by hand.

Is there any proper solution to this problem? If not, what is the generally accepted method to do this?

At the behest of many people, I am a student, but this question is geared to generally understanding the current methods of researcher evaluation, not necessarily just by students

lwl59438cuoly
  • 309
  • 1
  • 5
  • 8
    It'd probably be useful if you could explain the context more, i.e. why are you judging the researcher's status in the first place? In the context of choosing a research group it might be the wrong question to ask, for example. And the h-index has a large variability between fields and subfields, see e.g. this, so using it indiscriminately to compare two researchers' status is certainly overly simplistic. – Anyon Jul 13 '20 at 19:50
  • Was this not the question you posted about an hour ago and then deleted? – Solar Mike Jul 13 '20 at 20:04
  • 2
    @SolarMike No. I haven't posted and deleted a question recently. – lwl59438cuoly Jul 13 '20 at 20:25
  • @Anyon Surely there's some way to compare researchers then? – lwl59438cuoly Jul 13 '20 at 20:27
  • 4
    Yes, surely there are ways to compare researchers. Which way you choose depends on what your goals are, and for many of those goals you might not find any of the necessary information in a database. Search and tenure committees judge researchers all the time, but they don't rely on any one algorithm. Students judge them, too, in deciding who to work with. – Bryan Krause Jul 13 '20 at 20:32
  • @BryanKrause - I'm looking for an understanding of both how tenure committees, judges, researchers, or students would judge it. Are you sayinh this is too broad for one question and should be broken down into 4 different questions? – lwl59438cuoly Jul 13 '20 at 20:52
  • @lwl59438cuoly Probably each of those is also too broad even by itself, but before asking new questions, I'd spend an extensive time searching through existing Q&A here that addresses all of those topics. – Bryan Krause Jul 13 '20 at 20:59
  • Some example search terms to get you started might include: https://academia.stackexchange.com/search?q=tenure+decision https://academia.stackexchange.com/search?q=search+committee https://academia.stackexchange.com/search?q=choosing+advisor – Bryan Krause Jul 13 '20 at 21:00
  • 9
    Do you want to judge status or quality? Many quality researchers may not reach the status they deserve because of their country, race, gender, or any other sort of biases. – Azor Ahai -him- Jul 13 '20 at 21:01
  • @AzorAhai--hehim I guess I would want status. I was operating under the presumption that quality would eventually lead to status, but you do make an important point that I had not considered. I don't understand how I would even quantify status, so quality will do if status is not achievable – lwl59438cuoly Jul 13 '20 at 21:04
  • @lwl59438cuoly Well, I think status is much easier to answer, but it depends on whether you think it's useful if you actually want quality. – Azor Ahai -him- Jul 13 '20 at 21:16
  • @AzorAhai--hehim How would you judge the status? How would someone have good status but not be of high quality. I was under the assumption that there are 3 groups - "high status, high quality", "low status, high quality", and "low status, low quality" – lwl59438cuoly Jul 13 '20 at 21:22
  • 1
    @lwl59438cuoly The short answer is look at people who are asked to give talks at conferences --- especially keynote talks --- join panels, travel to universities to give small talks. However, this is biased towards White men (in the US). So you'd be less likely to identify high quality people by doing that. – Azor Ahai -him- Jul 13 '20 at 21:35
  • 1
    @lwl59438cuoly I guess the most obvious and extreme example of "high status, low quality" would be someone fraudulent like Schön before being exposed. – Anyon Jul 13 '20 at 21:35
  • 2
    I would not dichotimize - it's not high vs. low, but rather I'm saying status is not a perfect predictor of quality, and is biased in known ways – Azor Ahai -him- Jul 13 '20 at 21:36
  • 2
    I upvoted the question because I do suspect it is a question in many peoples' minds. But, at the same time, I do think it seriously missed the mark, even while being a natural sort of question. In any case, it's not that everyone is comparable by being assigned a single numerical score. (I know some people wish it could be so, perhaps administrators by number of external funding dollars...) – paul garrett Jul 13 '20 at 23:06
  • @paulgarrett I don't think OP is expecting it to be a fair comparison. The question explicitly states that OP will go through the top 3-5 and sort those by hand, so it simply has to be a "good enough" measure to eliminate the rest of the dataset – user760900 Jul 14 '20 at 16:56
  • 1
    You still haven't answered the question of why you would want to attempt this - since the answer obviously depends on what you need the information for. Personal opinion alert: academia is already fatally status-obsessed, why bother feeding into the grossness of the academic star system? – roger-reject Jul 15 '20 at 01:28

3 Answers3

8

Assuming the question is about status and not quality:

Job title is a strong indicator of status. The meaning of each title typically depends on the country. For the US, as an example, named professorships have the highest status, followed by professors and associate professors.

The status of the university is also a strong predictor of status of the researcher.

"Status" is not quantitative and attempting to rank status is not productive.

Anonymous Physicist
  • 98,828
  • 24
  • 203
  • 351
4

The best signifiers of status in a community will be being invited to give (plenary/keynote) at the (top) conferences for that community, and to win the prizes and awards that community bestows.

Having many highly cited papers will often correlate with that (publishing papers other people are interested in contributes to gaining status, high status makes your papers more likely to be cited), but this can be gamed; and there will be cases where people earn high status with a single remarkable accomplishment not translating well into citations, stuff like that.

As mentioned in the comments, quality and status are different things. As much as we'd like academia to be a pure meritocracy, it isnt. In a similar line, you won't be able to see who gets invited to give plenary talks, only who accepts. Some people being unable to travel (ill health, no funding/remote location, care responsibilities) will distort the picture.

Arno
  • 43,962
  • 8
  • 123
  • 169
  • 5
    Conference invitations in some fields (the ones I am familiar with) are biased towards men. – Anonymous Physicist Jul 13 '20 at 23:14
  • @AnonymousPhysicist I'd claim that that is a facet of women often not been given the status they deserve; not an issue with how well conference invitations work as indicator for status. – Arno Jul 14 '20 at 08:07
  • 2
    My comment was context for your answer, not a critique. – Anonymous Physicist Jul 14 '20 at 11:07
  • Your answer discusses post-invitation bias. I would upvote if you discussed invitation selection biases, per Anon's comments. – Azor Ahai -him- Jul 15 '20 at 20:58
  • Another issue here is the correct time-frame. Does someone with more invited talks in the past 20 years with no talks in the last five get ranked above someone with half the number of talks in the past 20 years but several talks in the past five? I think the question is impossible to answer without knowing for what purpose the ranking is to be used. – Terry Loring Jul 16 '20 at 18:14
-2

Use Scopus, because:

  • Sure it's maintained by Elsevier, but it also includes non-Elsevier papers.
  • Even if it does not have data from the last year, papers published in the last year probably have not have had time to accrue enough citations to change the h-index of well-established researchers.

Alternatively you could use a database such as Google Scholar or Web of Science.

Allure
  • 127,528
  • 50
  • 325
  • 493