 |
 |
Long post on the folly of any single system of measurement Thanks to Terry and Victor for a very revealing discussion. I find myself drawn to and worried by both positions. The UK research assessment exercise is arguably the granddaddy of these national reviews and it is instructive to look at some of the recent twists and turns that it has taken. Up till now it has always had a big element of direct peer review by expert panels although some natural science panels have tended to follow the journal rankings. In some smaller subjects there have been definite hints of a club operating as Terry describes although I think that has been overcome in the 2008 exercise by grouping panels under super panels to ensure consistent practice. When there was an attempt to introduce a greater reliance on metrics for future reviews the first thing everybody homed in on was journal rankings. Some disciplines thought that was OK but the arts and humanities were emphatic that this was not appropriate for the reasons that Victor gives. The fact that Terry's roots are in engineering and Victor's in the humanities is in line with this. Subsequently it has become a lot more murky. The UK government commissioned a report from University of Leiden in the Netherlands which flirted with the use of citation metrics and for a while there was a lot of debate about that. It's clear that citation metrics are not sufficiently developed to provide a universal measure across academia and also people are beginning to recognise that any mechanical system can be manipulated by clever academics to massage the results. Citation metrics may show who is at the top of the heap but the middle ranks can be pumped up by cronyism and various forms of self-citation or mutual citation. I was very excited to find my own citation rates on Google Scholar racing ahead recently but when I checked it was one guy in a different discipline who was repeatedly citing one of my papers in a very sloppy way to lend strength to a rather vague notion that I would not support. The UK government wants to find good measures of economic and social impact and knowledge transfer but that also is running into the sand as the research funding councils desperately look around for any exemplars or techniques they can use to identify and score these things. At the moment it feels as though every attempt to come up with sensible ways of measuring excellence or impact come round to the same conclusion that there are no reliable universal measures and a degree of direct peer review is inevitable. I support this as the UK system of selecting 4 "outputs" from the past 6 years for peer review by the RAE panel has encouraged people to focus on quality rather than quantity, even though university leaders sometimes miss the point about that. Finally I agree with Victor that peer reviewed journals are not the only form of valid "output" and a book or an important series of public lectures or some other contributions might be seen as very significant if the content is new and important, but I also agree with Terry that if you run an academic journal and don't have a peer review system you had better be ready to deal with accusations of cronyism. Personally I'd like us to abandon the double blind system in favour of open review which would really put quality and judgement in the spotlight but I can see that's a big step that we may not be ready for. However I can claim a great triumph for British Innovation. The RAE review has revealed that there is one reliable way to update and improve any system. If you don't think it's working and everybody is criticising you, why not change the name? The RAE is dead, long live the REF! (more information on all this at http://www.hefce.ac.uk/Research/assessment/reform/) best wishes from Sheffield, just finished editing and formatting 99 papers for DRS 2008 where I can confidently say that the refereeing and selection process was as good as you would hope for if you have 340 submissions to deal with in one big lump :o) best wishes Chris ********************* Professor Chris Rust Head of Art and Design Research Centre Sheffield Hallam University, S11 8UZ, UK +44 114 225 2706 [log in to unmask] www.chrisrust.net Every time I see an adult on a bicycle, I no longer despair for the future of the human race. - H. G. Wells