Friday, July 6, 2012

Reservations about the five year h-index

Recently I encountered a new metric, the "5 year h-index" which was being used to evaluate someone's research performance. Explicity, one counts the number h of papers published in the last 5 years that have been cited more than h times. Perhaps one might argue that this is a good metric for deciding whether to give someone a grant now. Afterall, just because 10 or 20 years ago they published highly cited papers does not mean that right now they are at the cutting edge. However, I do not agree.

I think this is a highly unreliable metric because there is significant noise. Except for a few rare exceptional papers, citations within a few years of publication will be low (1-10?). Hence, comparing two people with 5 year h-indexes, say one with a 6 and another with a 10, I would contend is meaningless.

Two of the most cited papers in Physical Review journals are the EPR (Einstein Podolsky Rosen) paper and Steven Weinberg's electro-weak interaction paper. The latter attracted about one citation per year for the first 5 years after publication! The former attracted about 10 citations in the first 2 years and then none for more than ten years!

When Jorge Hirsch introduced the h-index the whole point what to find some measure for a lifetime of scientific achievement. I still think it does provide a useful coarse-grained measure for that.

3 comments:

  1. It is also true that many papers with "pop" have little staying power once the trends of the day have passed. This index would lead to a world populated by Andy Warhol scientists, and not a Van Gough in the bunch.

    ReplyDelete
  2. Sounds better for ECRs than the usual h-index. On first thought at least.

    ReplyDelete
  3. This argument can apply in reverse as well. Many journals (and their editors) gloat over increases in Impact Factor, which measures an even shorter time the a 5 year h-index. Some journals have started to calculate their 5 year Impact Factor, which includes papers published up to 5 years ago. This seems more sensible than the regular Impact Factor.

    Having some kind of time horizon associated with an h-index or an Impact Factor has some value. If a person or a journal made a large impact 10-20 years ago but has done things with little impact (or nothing) since, this is meaningful information. The main question this raises for me is what the appropriate time horizon is.

    ReplyDelete