There have been many studies over the years (primarily authored by one C.
Oppenheim, but also by others) demonstrating a statistically significant
correlation between citation counts by academics returned for the RAE and
their department's eventual RAE scores. These studies cover hard science,
soft science and humanities; not sure if any studies have been done in
engineering subjects though.
Charles
Professor Charles Oppenheim
Department of Information Science
Loughborough University
Loughborough
Leics LE11 3TU
01509-223065
(fax) 01509-223053
----- Original Message -----
From: "Stevan Harnad" <[log in to unmask]>
To: <[log in to unmask]>
Sent: Wednesday, November 20, 2002 3:36 PM
Subject: Re: UK Research Assessment Exercise (RAE) review
> On Wed, 20 Nov 2002, [identity removed] wrote:
>
> > Dear Stevan,
> >
> > We are running a special report on the review of the RAE
> > and you were one of the people suggested as having more lively,
> > radical ideas about how research should be assessed in the future
[rather
> > than how the funding should be allocated]. I think HEFCE are a little
> > disappointed by the replies they have had so far, which seem
> > more about tweaking the current system than thinking 'out of the box'.
We
> > are asking a range of people what they would suggest. Would it be
possible
> > to talk to you about your ideas on this or perhaps you could email me
> > a few words on how you think the current system can be changed?
> >
> > Best wishes [identity removed]
>
> Happy to oblige.
>
> To summarize, the UK is in a unique position -- for the very reason that
> it is the only country with a national research assessment exercise like
> the RAE -- to do two very closely related things in concert, with three
> very likely and very positive outcomes:
>
> (i) It will give the UK RAE a far more effective and sensitive measure
> of research productivity and impact, at far less cost (both to the
> RAE and to the universities preparing their RAE submissions).
>
> (ii) Besides strengthening the assessment of UK research, it will
> also greatly strengthen the uptake and impact of UK research, by
> increasing its visibility, accessibility and uptake.
>
> (iii) At the same time, the UK RAE will thereby set an example to the
> rest of the world that will surely be emulated, in both respects:
> research assessment and research access.
>
> The proposal is quite simple, though I will spell it out as a series
> 20 closely connected points:
>
> (1) We already have an RAE, every 4 years.
>
> (2) It costs a great deal of time and energy (time and energy that
> could be used to actually do research, rather than preparing and
> assessing RAE returns) to prepare and assess, for both universities
> and assessors.
>
> (3) It is no secret that for most areas of research, the single most
> important and predictive measure of research impact is the so-called
> "impact factor" -- the number of times a work has been cited. This is
> a measure of the importance and uptake of that research.
>
> (4) The impact factor is used very indirectly in the RAE: Researchers each
> submit 4 publications for the 4-year interval, and these are (informally)
> weighted by the impact factor of the peer-reviewed journal in which
> they appeared. (For books or other kinds of publications, see below;
> in general, peer-reviewed journal or conference papers are the coin of
> the research realm, especially in scientific disciplines.)
>
> (5) If someone did a statistical correlation on the numerical outcome of
> the RAE, using the weighted impact factors of the publications of each
> department and institution, they would be able to predict the outcome
> ratings quite closely. (No one has done this exact statistic, because
> the data are implicit rather than explicit in the returns, but it could
> be done, and it would be a good idea to do it, just to get a clear
> indication of where the RAE stands right now, before the simple reforms
> I am recommending.)
>
> (6) There is no reason why the RAE should be based only on the impact
> factors of 4 publications per researcher, nor why it should be weighted
> by the impact factor of the journal in which it appeared, rather than on
> the impact of each publication itself. (On average the two will agree, but
> there is no reason to rely on blunt-instrument averages if we can use a
> sharper instrument: A researcher's individual paper may have a much
> higher -- or lower -- impact than the average impact of the journal in
> which it appears.)
>
> (7) Nor is there any reason why the RAE should be done, with great
> effort and expense, every 4 years!
>
> (8) Since the main factor in the RAE outcome ratings is research impact,
> there is no reason whatsoever why research impact should not be
> continuously assessed -- and directly, rather than indirectly, via the
> the true impact factor of the publication (or the author!), rather
> than merely the journal's average impact factor.
>
> (9) And there is now not only a method to (a) continuously assess full
> UK research impact, and not only get this be done (b) incomparably more
> cheaply and less effortfully for all involved, while at the same time
> being (c) more sensitive and accurate in estimating the true impact of
> the research, but doing the RAE this new way will also have a dramatic
> effect on the magnitude of UK research impact itself, (d) increasing
> its visibility, usage, citation and productivity dramatically, simply
> by maximizing it accessibility.
>
> (10) The method in question is to implement the RAE henceforth online
> only, and the only two critical elements are (1) the submission of a
> RAE-standardized online CV by each researcher and (2) a link in each CV
> between every published paper -- books discussed separately below -- and
> the full digital text of that paper in that researcher's university
> Eprint Archive (an online archive of that institution's peer-reviewed
> research output).
> http://www.eprints.org/self-faq/#institution-facilitate-filling
> (See the free, open-source software we have developed
> at Southampton to allow universities to create their own institutional
> Eprint Archives: http://software.eprints.org/ )
>
> (11) Currently, university peer-reviewed research output -- funded
> by government research grants, the researcher's time paid for by the
> researcher's institution -- is given, free, by all researchers, to the
> peer-reviewed journals in which it appears.
>
> (12) The peer-reviewed journals in turn perform the peer-review, which
> assesses and improves the quality of the research (this is one of the
> indirect reasons that the RAE depends on peer-reviewed journal
> publications).
>
> (13) There is a hierarchy of peer-reviewed journals, from those with the
> highest quality standards (and hence usually the highest rejection rates
> and impact factors) at the top, grading all the way down to the
> lowest-quality journals at the bottom.
>
> (14) The peer-reviewers referee for free; they are just the researchers
> again, wearing other hats.
>
> (15) But the it costs the journals something to implement the peer
> reviewing. (Estimates are that it costs $200-$500 per paper.)
>
> (16) Partly because of the cost of peer review, but mostly because of the
> much larger cost of print-on-paper and its dissemination, journals charge
> tolls (subscriptions. licenses, pay-per-view) for access to researchers'
> research output (even though the researchers gave them the research for
> free).
>
> (17) The result of the access-tolls is a great loss of potential research
> impact, because most institutions cannot afford the access tolls to most
> peer-reviewed journals (there are 20,000 in all, across disciplines),
> but only to a small and shrinking proportion of them.
>
> (18) Hence the second dramatic effect of revising the RAE to make it
> online continuous assessment based on the institutional self-archiving
> of all UK peer-reviewed research output is that it will make all that UK
> research accessible to all would-be users worldwide whose access is
> currently blocked by access-toll-barriers.
> http://www.ecs.soton.ac.uk/~harnad/Tp/resolution.htm#4.2
>
> (19) The UK full-text peer-reviewed research archives will not only be
> continuously accessible to all potential users, but the access will be
> continuously assessable, in the form not only of continuously updated
> impact factors based on the classical measure of impact, which is
> citations, but usage will also be measured at earlier stages than
> citation, namely downloads ("hits"). And many other powerful online
> measures of research productivity and impact will develop around this UK
> research corpus, increasing the sensitivity and predictiveness of the
> RAE analyses more and more.
> (See the online impact-measuring scientometric search engines we have
> developed at Southampton: http://citebase.eprints.org/cgi-bin/search
> and http://opcit.eprints.org )
>
> (20) And all that is needed for this is for RAE to revert to online
> submissions, requiring online CVs linked to the full-text draft
> of each paper in the researcher's institutional Eprint Archive.
> http://www.eprints.org/self-faq/#research-funders-do
> Assessment engines like citebase can then be used by RAE to derive
> ever richer and more effective measures of research productivity
> and impact, available to the RAE continuously. And institutions
> could continuously monitor and improve their own research productivity
> and impact, using the same measures. And the rest of the world could see
> and emulate the system, and its measurable effects of research
> visibility, uptake and impact.
>
> Just a few loose ends: Books are often not give-aways, as peer-reviewed
> research is, so this solution does not apply as well to the assessment
> of book output -- but this was a problem even in the old system, because
> impact measures are not as readily available or widely used for books.
> The new system will strengthen the RAE and its accuracy and fairness in
> all sectors except books. And even with books there is the option
> (especially with esoteric monographs that produce virtually no royalty
> revenue) to put them in the Institutional Eprint Archives too. And even
> if the book itself is not accessible online, the citation of books by
> the online peer-reviewed publications will be a measurable and usable
> estimate of their impact.
>
> The UK is uniquely placed to move ahead with this and lead the world,
> because the RAE is already in place. But we need to move fast,
> because other countries are getting the idea too: The Netherlands has
> no formal RAE yet, but it is about to implement a national system of open
> research archiving for all of its universities called DARE:
> http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/2356.html
> It is just a matter of time before they realize that a marriage between
> a national network of DARE-style institutional Eprint Archives and a
> national RAE-style research assessment exercise makes a natural, indeed
> an optimal combination. If/when they do, it will be they, and not the
> UK, who lead the rest of the world toward this natural solution.
>
> But although the naturalness and optimality -- indeed the inevitability
> -- of all this is quite transparent, it is a fact that research culture
> is slow to change of its own accord, even in what is in its own best
> interests. But that is precisely why we have funding councils and
> research assessment: To make sure that researchers do what is best for
> themselves, and for research, and hence for the supporters (and
> beneficiaries) of research, namely, tax-paying society: Institutional
> self-archiving of research output, for the sake of maximizing research
> access and impact has been much to slow in coming, even though it has
> already been within reach for several years. The UK and the RAE are now
> in a position to lead the world research community to the optimal and
> the inevitable.
> http://www.eprints.org/self-faq/#research-funders-do
>
> We at Southampton, meanwhile, are continuing to try to do our bit to
> hasten the optimal/inevitable for research and researchers. We are
> planning to harvest all the metadata form the submissions to RAE 2001
> http://www.hero.ac.uk/rae/submissions/ into RAEprints, a "meta-archive,"
> that is meant to demonstrate what RAE returns would like if this
> recommendation were adopted. Of course RAEprints (i) will contain only
> four papers per researcher, rather than their full peer-reviewed research
> output, (ii) it will only contain the metadata for those papers (author,
> title, journal-name), not the full-text and the all-important references
> cited. But we will also try to enhance the demo by adding as much of
> this missing data as we can find on the Web, so as to at least give a
> taste of the possibilities: Using paracite http://paracite.eprints.org/
> an on-line citation-seeker that goes out and tries to find peer-reviewed
> full-text papers on the web, we will "stock" RAEprints with as much as
> we can find -- and then we will invite all the RAE 2001 papers to add
> their full-texts too!
>
> But we can't do it all alone. We hope HEFCE and RAE will put their full
> weight behind progress toward this outcome, so beneficial to so many, in
> so many ways.
>
> Best wishes,
>
> Stevan
>
> -------------
>
> Date: Fri, 25 Oct 2002 19:06:55 +0100 (BST)
> From: Stevan Harnad <[log in to unmask]>
>
> This is a response to the HEFCE "invitation to contribute"
> recommendations for restructuring the RAE:
>
> I have written two papers on how the RAE might be greatly improved in
> its assessment accuracy and at the same time made far less effortful
> and costly -- while (mirabile dictu) doing a great indirect service
> to research and researchers, both in the UK and in the rest of the
> scholarly/scientific world as well:
>
> Harnad, S. (2001) "Research access, impact and assessment." Times
Higher
> Education Supplement 1487: p. 16.
> http://cogprints.soton.ac.uk/documents/disk0/00/00/16/83/index.html
>
> Harnad, S. (2001) The Self-Archiving Initiative. Nature 410: 1024-1025
> http://www.nature.com/nature/debates/e-access/Articles/harnad.html
>
> If you wish to see what the RAE would look like if UK research output
> were continually accessible online, and hence continuously assessable,
see:
> http://citebase.eprints.org/
>
> To see how the RAE could help hasten this outcome (which is in any case
> optimal and inevitable), see:
> http://www.eprints.org/self-faq/#research-funders-do
>
> We at Southampton are currently harvesting the RAE submissions data
> and putting them in an Eprint Archive to provide a "demo" of the sorts of
> possibilities an online, open-access research corpus opens up for
> research visibility, accessibility, uptake, usage, citation, impact and
> assessability.
>
> Best wishes,
>
> Stevan Harnad
>
>
|