Long post on the folly of any single system of measurement
Thanks to Terry and Victor for a very revealing discussion. I find
myself drawn to and worried by both positions.
The UK research assessment exercise is arguably the granddaddy of these
national reviews and it is instructive to look at some of the recent
twists and turns that it has taken. Up till now it has always had a big
element of direct peer review by expert panels although some natural
science panels have tended to follow the journal rankings. In some
smaller subjects there have been definite hints of a club operating as
Terry describes although I think that has been overcome in the 2008
exercise by grouping panels under super panels to ensure consistent
practice.
When there was an attempt to introduce a greater reliance on metrics for
future reviews the first thing everybody homed in on was journal
rankings. Some disciplines thought that was OK but the arts and
humanities were emphatic that this was not appropriate for the reasons
that Victor gives. The fact that Terry's roots are in engineering and
Victor's in the humanities is in line with this.
Subsequently it has become a lot more murky. The UK government
commissioned a report from University of Leiden in the Netherlands which
flirted with the use of citation metrics and for a while there was a lot
of debate about that. It's clear that citation metrics are not
sufficiently developed to provide a universal measure across academia
and also people are beginning to recognise that any mechanical system
can be manipulated by clever academics to massage the results.
Citation metrics may show who is at the top of the heap but the middle
ranks can be pumped up by cronyism and various forms of self-citation or
mutual citation. I was very excited to find my own citation rates on
Google Scholar racing ahead recently but when I checked it was one guy
in a different discipline who was repeatedly citing one of my papers in
a very sloppy way to lend strength to a rather vague notion that I would
not support.
The UK government wants to find good measures of economic and social
impact and knowledge transfer but that also is running into the sand as
the research funding councils desperately look around for any exemplars
or techniques they can use to identify and score these things.
At the moment it feels as though every attempt to come up with sensible
ways of measuring excellence or impact come round to the same conclusion
that there are no reliable universal measures and a degree of direct
peer review is inevitable. I support this as the UK system of selecting
4 "outputs" from the past 6 years for peer review by the RAE panel has
encouraged people to focus on quality rather than quantity, even though
university leaders sometimes miss the point about that.
Finally I agree with Victor that peer reviewed journals are not the only
form of valid "output" and a book or an important series of public
lectures or some other contributions might be seen as very significant
if the content is new and important, but I also agree with Terry that if
you run an academic journal and don't have a peer review system you had
better be ready to deal with accusations of cronyism. Personally I'd
like us to abandon the double blind system in favour of open review
which would really put quality and judgement in the spotlight but I can
see that's a big step that we may not be ready for.
However I can claim a great triumph for British Innovation. The RAE
review has revealed that there is one reliable way to update and improve
any system. If you don't think it's working and everybody is criticising
you, why not change the name? The RAE is dead, long live the REF!
(more information on all this at
http://www.hefce.ac.uk/Research/assessment/reform/)
best wishes from Sheffield, just finished editing and formatting 99
papers for DRS 2008 where I can confidently say that the refereeing and
selection process was as good as you would hope for if you have 340
submissions to deal with in one big lump :o)
best wishes
Chris
*********************
Professor Chris Rust
Head of Art and Design Research Centre
Sheffield Hallam University, S11 8UZ, UK
+44 114 225 2706
[log in to unmask]
www.chrisrust.net
Every time I see an adult on a bicycle, I no longer despair for the
future of the human race. - H. G. Wells
|