33

Like many academics with an active research program, I am asked to review many papers. Borrowing an idea from peer-to-peer file sharing companies, I try to keep a rough sense of my recent "ratio" of reviews I have done on other's papers over reviews my papers have received. I try to make sure I have a ratio that is consistently above 1:1. Since many of my papers (and presumably the papers I review) are co-authored, this seems very generous to me.

How many reviews should I be doing? How many is normal? Is a ratio an effective way to keep track of one's reviews? Is a minimum 1:1 ratio appropriate and defensible?

mako
  • 12,831
  • 1
  • 46
  • 76

5 Answers5

25

I have wondered about the same question. One thing that I've noticed is that there is an incredible amount of variation in the amount of reviewing done. I have two good friends who are strong, active mathematicians. The last time I spoke to them they were each at least six years past their PhD and had never reviewed a single paper. This seems bad, but it's not their fault: they are not being asked.

I suspect that the lack of coordination among journals in the same field causes the same small number of people to be asked to referee papers over and over again. I am not a journal editor, but I have begun to wonder how editors choose referees: I am someone who is getting (what I think is!) a large number of referee requests. I submit maybe 2-4 papers per year. Not so long ago I used to get that number of referee requests per year, and they were all from people who knew me directly and pertained to topics related to my thesis work or to the areas in which I have written multiple papers. Now I get at least a dozen referee requests a year, and in addition to the above I get some very strange-looking ones. Not so long ago I got asked to referee a paper in (what I think is!) control theory. This was so baffling that I assumed it was some kind of email crossed wires situation and asked the editor about it. It turned out that the paper on real analytic solutions to Bezout equations had been sent to me because of a question on math.SE that I answered, in which I explained why the ring of analytic functions on the complex plane is a Bezout domain. This is distressing: I can see why google would make a connection like this, but in the realm of academic experts, is it not obvious that we are on different sides of mathematics? Experiences like this make me wonder who is refereeing papers.

I'm not sure that counting papers is the right way to gauge whether you are pulling your weight referee-wise. As I mentioned in a comment to another answer, many -- I think most -- papers are refereed by more than one person, either simultaneously for the same journal or sequentially, as many (most?) papers are submitted to more than one journal before they are accepted. Here are some things that I have thought about:

1) It cannot be appropriate for everyone to referee the same number of papers. More experienced people should referee more papers, but also should referee more important / difficult papers. People who have permanent / senior positions in academia (especially: tenure) should have more obligations than those who are struggling to stay in the profession.

2) Not all papers take the same amount of time to referee: not even close, actually. I try to make a point of quickly processing especially short and simple papers: I probably do 3-5 of those each year, and I often process them in a week or even a day. But it would be silly to equate this kind of referee work with the long, difficult papers that you spend weeks or months working on, perhaps having to do considerable outside reading in order to adequately evaluate. Doing 2-3 substantial papers a year is what really keeps me busy as a referee. I suppose I try to keep my eye on the total amount of time I've spent refereeing rather than on the number of papers, although I admit that I don't do this in any formalized way.

3) Unless a paper is clearly of the "one day" variety as above, I try not to let my queue exceed 2 papers. This is also out of respect to the authors: if I have to read two papers before I get to yours, then you'll be waiting for at least six months, probably more. If at any given time every sufficiently experienced/qualified academic had either 0 or 1 papers to referee and made a point of processing papers faster than they submit them, then things would work relatively smoothly, I suppose.

Pete L. Clark
  • 130,631
  • 25
  • 361
  • 522
  • If you are submitting 4 papers per year and getting at least 2 reviews for each paper, then 12 papers is about the amount that you need to be reviewing to pay back the reviewing and editing time that is being put into your submissions. See My answer below for more on why. As to how editors select reviewers, we look for reviewers who are already in our database and have keyword matches with the submitted paper, or else we look for people who have published on similar topics in recent years. If your reviews are good and timely, you will be asked more often. – Significance Feb 26 '16 at 08:20
  • 1
    I think that point 2 is an excellent one: simply counting the number of papers reviewed is pointless. Measuring the time spent on the reviews is much more meaningful. – DCTLib Feb 26 '16 at 09:29
15

Speaking as an editor and conference organizer, good and reliable reviewers are hard to find. For somebody to be a good reviewer, in addition to being in the right discipline, they must be:

  1. A strong enough scientist that I trust their opinion
  2. Capable of expressing their analysis of a paper clearly and constructively
  3. Good enough at time-management to return the review rapidly
  4. Not too overloaded with other responsibilities to accept

Lots of authors fail these tests, especially the first one---anybody who has spent significant time reviewing knows that there are a lot of very bad papers being written. Moreover, papers that are rejected often get revised and sent elsewhere, so they may require multiple sets of reviewers. All of this adds up to the following: if you are a good and trusted reviewer in your community, you will be in high demand, and it's reasonable to have a ratio of greater than 1:1.

For example, in the last three years, excluding grant and tenure case reviews, according to my records I have reviewed 31 papers in 2012, 24 papers in 2013, and 30 papers so far in 2014 (with a few more committed to return before the end of the year). This puts my ratio well above 1:1, with the actual value mainly depending on fluctuations in the denominator.

In sum, I recommend viewing reviewing not in terms of ratios, but in terms of how much time you are comfortable devoting to quality reviews for venues that you wish to support. Don't accept review requests from journals or conferences that you do not respect: there are many low-quality publications that can waste your time. Also, don't accept more reviewing than you can handle, even if that means your ratio is lower: it's better to give one well-thought review than two hasty and shallow reviews that may miss important points and piss off the authors. Do respond yes or no to every review request promptly: otherwise, you are making a problem for an editor who doesn't know whether they need to recruit more reviewers or not.

jakebeal
  • 187,714
  • 41
  • 655
  • 920
13

You could count each n-author paper you are on as 1/n of a paper. Thus each time you submit for publication a 2-author paper, a 3-author paper and a 6-author paper, it's time to referee another paper.

If the journal uses more than one referee per paper, adjust up accordingly.

In some sense this must be what most people do, on average.

Edit: One could ask the editors how many referees they use per paper. Better yet, the editors should volunteer that information.

Bjørn Kjos-Hanssen
  • 3,168
  • 18
  • 26
  • 2
    I don't know of any evidence that suggests that this is what people do on average. My sense is that some people review tons of papers and many people (e.g., graduate students) submit papers but are rarely asked to review. – mako Sep 27 '14 at 22:07
  • 10
    @BenjaminMakoHill: On average, people do exactly as many reviews as they receive (scaled by the number of coauthors). Every review that someone receives comes from someone. – Jukka Suomela Sep 27 '14 at 22:36
  • 2
    The averages must be exactly the same if we average over all people who have ever submitted or reviewed a paper, but not necessarily over people currently active in the field. (For example, it's possible that people who leave the field have on average submitted more papers than they have reviewed.) – Anonymous Mathematician Sep 27 '14 at 22:57
  • @JukkaSuomela. If we treat the groups of all authors and reviewers as a single pool to average over, that is correct although I'm not sure that's fair. I only meant to suggest that if we were look at the distribution of reviews over reviewers, I suspect to see a very skewed distribution. In any case, I'm less interested in the mathematical statements about averages than I am in a discussion of what someone should do. – mako Sep 27 '14 at 22:59
  • @BenjaminMakoHill: I fail to see how the "recommended" number could be below the average number. Otherwise, if everyone followed the recommendation, the system would collapse. – Jukka Suomela Sep 27 '14 at 23:16
  • @JukkaSuomela. Yes, but it could certainly be above. – mako Sep 27 '14 at 23:36
  • 1
    The fact that many (most?) papers are reviewed by more than one person (either by the same journal or for different journals) does not seem to be taken into account in the analysis of this answer. This makes a figure of reviewing twice as many papers as you submit look fairer/safer from a "pulling your average weight" perspective. – Pete L. Clark Sep 28 '14 at 00:06
  • Added: I missed the sentence "If the journal uses more than one referee per paper, adjust up accordingly." Sorry. But some journals use only one referee, some use several, and I don't know all the statistics, so it's hard to adjust. Also: this does not take into account that the same paper may be refereed by several different journals. If I say that I think on average each paper is refereed by 3 people and you say it's 2, then we're differing significantly in how much work we think any one person should do. So does everyone just solve the Fermi problem in their own way? – Pete L. Clark Sep 28 '14 at 00:36
  • 1
    @PeteL.Clark: "…so it's hard to adjust…": Actually, all this is trivial to adjust. You do not need to consider "average papers" or estimate anything. Just consider those papers that you review and those papers that you submit. Everyone knows (at least in principle) how many reviews they have personally written, and everyone knows how many (coauthor-normalised) reviews they have received in total (including everything: re-submissions, multiple reviews per submission, etc.). Just keep track of these two numbers and make sure that in the long run their difference is positive. – Jukka Suomela Sep 28 '14 at 07:07
  • 5
    (Just to clarify. When you review a paper, the math goes like this: "this counts as 1 review written by me". And when you submit a paper with n coauthors and receive the reports and see that there are k reviews, the math goes like this: "this counts as k/n reviews received by me". You do the same thing regardless of whether your paper was rejected or accepted. There is no need to get any additional information from the editors.) – Jukka Suomela Sep 28 '14 at 07:14
  • @Jukka: Just because you see k reports does not mean that only k referees were enlisted to review the paper. I have had papers both accepted and rejected without receiving any referee reports. Also, the business about the coauthors and fractions assumes that your coauthors are doing the same amount of refereeing as you are, which if they are much more junior or senior than you may not be the case. Many of my coauthors are graduate students who do little or no refereeing. – Pete L. Clark Sep 28 '14 at 08:11
  • 5
    But I agree with the gist of what you're saying: trying to compute the average is needlessly complicated. You can probably get a pretty good idea of how many referees you "use" per year and make sure that you stay ahead of that. How far ahead you want to be is still a judgment call, as I talk about in my answer. – Pete L. Clark Sep 28 '14 at 08:15
  • @BenjaminMakoHill, just add up the papers submitted and the papers reviewed, the sums must match. Handshaking lemma of graph theory in disguise ;-) – vonbrand Feb 27 '16 at 00:32
0

Four times as many as you submit as corresponding author.

My reasoning is:

  • Most journals solicit between two and four referee reports for each paper that is sent out for review. Add to this the time of the volunteer editorial board member who is probably handling the manuscript.
  • Some academics do not do their duty to the community and decline all review requests, so the rest of us have to make up for them.
  • By my reckoning, this works out to about 4 x the number of papers you submit being a fair load.
Significance
  • 6,054
  • 1
  • 21
  • 40
  • 2
    "Most journals solicit between two and four referee reports for each paper that is sent out for review." This is highly subject-dependent. In my field -- mathematics -- the most common number of referees is one. I have submitted over 30 papers for publication and never gotten more than two. – Pete L. Clark Feb 26 '16 at 09:43
  • 3
    By the way, if you want to be quantitative about it (which, as I say in my answer, I don't necessarily think is the best way to go), then how is the number of coauthors taken into account in your reckoning? – Pete L. Clark Feb 26 '16 at 10:02
  • The number of co-authors is irrelevant if you count only the number of papers that you submit. You, personally, as corresponding author. I'm really surprised to hear that one review is sufficient in some fields. That must speed up the review process a lot, and make responding to review comments much easier (no navigating conflicting reviewer opinions)! The journal for which I am an editor prefers 4 reviews and requires at least 3 for each manuscript. – Significance Feb 26 '16 at 22:03
  • "That must speed up the review process a lot, and make responding to review comments much easier (no navigating conflicting reviewer opinions)!" Well, one frequently gets referee reports in less than a year, but not always. I hope, for your sake, that that doesn't sound that fast to you. – Pete L. Clark Feb 26 '16 at 22:21
  • I missed the "corresponding author" part of your answer, in part because it didn't appear in your comment. No one I know would interpret "I submit 2-4 papers per year" as "I submit 2-4 papers per year as corresponding author," since in my field "corresponding author" means precisely "the person who sends the email to the journal / fills out the online form". I know of no reason not to divide corresponding author duties roughly equally among authors, but on the other hand I have no guarantee of it. Some people are undoubtedly corresponding authors more or less often than their coauthors. – Pete L. Clark Feb 26 '16 at 22:25
  • Should someone who doesn't enjoy filling out online forms and therefore has her coauthors do that really be relieved of all refereeing duties? Or if e.g. a three author paper needs to be shopped around to three journals before it gets published, probably the same person will be corresponding author all three times. Shouldn't this count as one referee job per author, not three jobs for the corresponding author? So your reckoning doesn't seem very accurate to me. – Pete L. Clark Feb 26 '16 at 22:27
  • Corresponding author has high significance in some fields. – jakebeal Feb 27 '16 at 00:51
  • @PeteL.Clark Fair enough. You could use "first author" instead of "corresponding author" if you prefer. Or simply divide the number of papers on which you are an author by the average number of co-authors on those papers. I'm used to the corresponding author being the first author, and the first author being the one who has done most of the work and prompted the submission of the paper, but I agree this isn't always the case. – Significance Feb 27 '16 at 00:59
  • @Significance: Please remember there are several academic fields -- like mathematics -- in which "first author" means "the author whose last name comes first alphabetically". This would be a strange way of assigning referee duties. If you only intend your answer to apply to certain academic fields, that 's fine but please say so. On the other hand, there is already an answer here (Kjos-Hanssen's) giving a system which works independently of any implicit cultural assumptions about coauthorship. Why do you need to list one number that is intended to work for everyone but actually doesn't? – Pete L. Clark Feb 27 '16 at 03:25
  • @PeteL.Clark the comment to which you are replying already acknowledged this point (that not all fields follow the same practices). Kjos-Hanssen's answer is now generic and sufficient, as it has been edited, but the original version, which was there when I first commented, assumed only one reviewer per paper, which is unlikely in most fields. My answer was intended to make the same point more clearly and simply: i.e. pay back what is done for you. – Significance Feb 27 '16 at 03:45
  • @Significance: Kjos-Hanssen's answer has not been changed for more than a year before you joined the site. "Pay back what is done for you" is a good, simple answer but is certainly not equivalent to "Referee four times as many papers as you submit as a corresponding author." – Pete L. Clark Feb 27 '16 at 03:56
  • You don't seem to acknowledge that there are entire academic fields in which your answer is a huge overestimate. For instance, I am supervising a postdoc who in the last year was the "corresponding author" on one paper and the referee for one paper. Number of referees involved in getting her paper refereed: one. You are saying that to pull her weight she should be doing four times as much refereeing. But four times as much refereeing for a careful, junior mathematician would put her out of business. So my comments are based on imagining someone like her reading your answer. – Pete L. Clark Feb 27 '16 at 03:58
  • @PeteL.Clark I acknowledge, and have now acknowledged three times that there are academic fields in which the average number of reviews is lower and/or the corresponding author is not the best way of assigning responsibility. I am certainly not saying that your postdoc is not pulling her weight. I stated the assumptions in my original answer clearly so that it was clear in which circumstances they did not apply. I don't believe that you sincerely think we have a substantive disagreement here, so I will not be drawn further. Yes, you may reply and have the last word. – Significance Feb 27 '16 at 04:04
  • 1
    "I stated the assumptions in my original answer clearly so that it was clear in which circumstances they did not apply." You give a figure unconditionally, not under certain hypotheses. "I don't believe that you sincerely think we have a substantive disagreement here." I sincerely think we have a substantive disagreement: you are giving advice that would be harmful if followed by some, and I disagree that it will be obvious who should follow your advice (and you certainly have not said). But I do think that the comments and voting will help readers not be misled, so it is probably okay. – Pete L. Clark Feb 27 '16 at 04:22
0

First of all, I totally agree with the answer of Pete L. Clark.

However, I do not agree with the mentality that thinks: "Reviewing a paper is JUST doing a favour to the community". We all read papers to stay on top of our fields, get a better understanding of the world around us and to get new insights. I personally read several papers every month. Nonetheless, every paper that I read, I must maintain my critical view. So, there is no harm in reviewing many papers.

When you receive a paper that seems interesting to you, you can decide to review it (of course there is a burden of writing down your thoughts, which is negligible). If the paper is not interesting, you can just decline the review request and move on.

Community
  • 1
orezvani
  • 1,227
  • 11
  • 16