A poor metric of research quality

Australia has just begun their own version of the UK Research Assessment Exercise (RAE). Ours is called the ERA (Excellence in Research for Australia).

[Although I show my age and the fact that I grew up reading Time magazine because I always think of the ERA as the Equal Rights Amendment].

Like the RAE the reporting requirements and the associated bureaucracy are onerous. Whether the ERA will end up having the same profound implications as the RAE in the UK remains to be seen.

As part of the data gathering I was required to provide the following statistic:

the fraction of my publications during 2003-2008 that involved co-authors who were not from my institution.

(It turned out to be 40 per cent).

I did not get the opportunity to ask whether we wanted this number to be high or low.

I realise this is just one of many metrics that will be used to judge research quality. Nevertheless, it seems to me that this metric is a rather problematic measure of research quality. I suspect most good researchers will be somewhere in the range of 30 to 70 per cent. But, neither a high nor low number necessarily seems good to me.

For example, this number could be high because the researcher is just a bit player in a much larger team effort led but other institutions. Or one still only publishes papers with ones Ph.D advisor (and on the same topic!).

On the other hand, a high number may result because the researcher always recruits world leaders with complimentary expertise for projects. I think this is increasingly important for basic materials science.

But a low number could be either good or bad, too. For example, low number may be because a researcher is the world leader in a particular area and they have built up a self-contained team at their own institution.

On the other hand, a low number may be a result of mediocrity: e.g. no one wants to collaborate with them, or they settle for doing everything themselves rather than collaborate with world leaders with complementary expertise.

I also think whether this number is a good metric for quality will vary significantly with the research project, field and the level of specialisation and the extent it is multi-disciplinary.