On Wed, 20 Nov 2002, [identity removed] wrote:
> Dear Stevan,
>
> We are running a special report on the review of the RAE
> and you were one of the people suggested as having more lively,
> radical ideas about how research should be assessed in the future [rather
> than how the funding should be allocated]. I think HEFCE are a little
> disappointed by the replies they have had so far, which seem
> more about tweaking the current system than thinking 'out of the box'. We
> are asking a range of people what they would suggest. Would it be possible
> to talk to you about your ideas on this or perhaps you could email me
> a few words on how you think the current system can be changed?
>
> Best wishes [identity removed]
Happy to oblige.
To summarize, the UK is in a unique position -- for the very reason that
it is the only country with a national research assessment exercise like
the RAE -- to do two very closely related things in concert, with three
very likely and very positive outcomes:
(i) It will give the UK RAE a far more effective and sensitive measure
of research productivity and impact, at far less cost (both to the
RAE and to the universities preparing their RAE submissions).
(ii) Besides strengthening the assessment of UK research, it will
also greatly strengthen the uptake and impact of UK research, by
increasing its visibility, accessibility and uptake.
(iii) At the same time, the UK RAE will thereby set an example to the
rest of the world that will surely be emulated, in both respects:
research assessment and research access.
The proposal is quite simple, though I will spell it out as a series
20 closely connected points:
(1) We already have an RAE, every 4 years.
(2) It costs a great deal of time and energy (time and energy that
could be used to actually do research, rather than preparing and
assessing RAE returns) to prepare and assess, for both universities
and assessors.
(3) It is no secret that for most areas of research, the single most
important and predictive measure of research impact is the so-called
"impact factor" -- the number of times a work has been cited. This is
a measure of the importance and uptake of that research.
(4) The impact factor is used very indirectly in the RAE: Researchers each
submit 4 publications for the 4-year interval, and these are (informally)
weighted by the impact factor of the peer-reviewed journal in which
they appeared. (For books or other kinds of publications, see below;
in general, peer-reviewed journal or conference papers are the coin of
the research realm, especially in scientific disciplines.)
(5) If someone did a statistical correlation on the numerical outcome of
the RAE, using the weighted impact factors of the publications of each
department and institution, they would be able to predict the outcome
ratings quite closely. (No one has done this exact statistic, because
the data are implicit rather than explicit in the returns, but it could
be done, and it would be a good idea to do it, just to get a clear
indication of where the RAE stands right now, before the simple reforms
I am recommending.)
(6) There is no reason why the RAE should be based only on the impact
factors of 4 publications per researcher, nor why it should be weighted
by the impact factor of the journal in which it appeared, rather than on
the impact of each publication itself. (On average the two will agree, but
there is no reason to rely on blunt-instrument averages if we can use a
sharper instrument: A researcher's individual paper may have a much
higher -- or lower -- impact than the average impact of the journal in
which it appears.)
(7) Nor is there any reason why the RAE should be done, with great
effort and expense, every 4 years!
(8) Since the main factor in the RAE outcome ratings is research impact,
there is no reason whatsoever why research impact should not be
continuously assessed -- and directly, rather than indirectly, via the
the true impact factor of the publication (or the author!), rather
than merely the journal's average impact factor.
(9) And there is now not only a method to (a) continuously assess full
UK research impact, and not only get this be done (b) incomparably more
cheaply and less effortfully for all involved, while at the same time
being (c) more sensitive and accurate in estimating the true impact of
the research, but doing the RAE this new way will also have a dramatic
effect on the magnitude of UK research impact itself, (d) increasing
its visibility, usage, citation and productivity dramatically, simply
by maximizing it accessibility.
(10) The method in question is to implement the RAE henceforth online
only, and the only two critical elements are (1) the submission of a
RAE-standardized online CV by each researcher and (2) a link in each CV
between every published paper -- books discussed separately below -- and
the full digital text of that paper in that researcher's university
Eprint Archive (an online archive of that institution's peer-reviewed
research output).
http://www.eprints.org/self-faq/#institution-facilitate-filling
(See the free, open-source software we have developed
at Southampton to allow universities to create their own institutional
Eprint Archives: http://software.eprints.org/ )
(11) Currently, university peer-reviewed research output -- funded
by government research grants, the researcher's time paid for by the
researcher's institution -- is given, free, by all researchers, to the
peer-reviewed journals in which it appears.
(12) The peer-reviewed journals in turn perform the peer-review, which
assesses and improves the quality of the research (this is one of the
indirect reasons that the RAE depends on peer-reviewed journal
publications).
(13) There is a hierarchy of peer-reviewed journals, from those with the
highest quality standards (and hence usually the highest rejection rates
and impact factors) at the top, grading all the way down to the
lowest-quality journals at the bottom.
(14) The peer-reviewers referee for free; they are just the researchers
again, wearing other hats.
(15) But the it costs the journals something to implement the peer
reviewing. (Estimates are that it costs $200-$500 per paper.)
(16) Partly because of the cost of peer review, but mostly because of the
much larger cost of print-on-paper and its dissemination, journals charge
tolls (subscriptions. licenses, pay-per-view) for access to researchers'
research output (even though the researchers gave them the research for
free).
(17) The result of the access-tolls is a great loss of potential research
impact, because most institutions cannot afford the access tolls to most
peer-reviewed journals (there are 20,000 in all, across disciplines),
but only to a small and shrinking proportion of them.
(18) Hence the second dramatic effect of revising the RAE to make it
online continuous assessment based on the institutional self-archiving
of all UK peer-reviewed research output is that it will make all that UK
research accessible to all would-be users worldwide whose access is
currently blocked by access-toll-barriers.
http://www.ecs.soton.ac.uk/~harnad/Tp/resolution.htm#4.2
(19) The UK full-text peer-reviewed research archives will not only be
continuously accessible to all potential users, but the access will be
continuously assessable, in the form not only of continuously updated
impact factors based on the classical measure of impact, which is
citations, but usage will also be measured at earlier stages than
citation, namely downloads ("hits"). And many other powerful online
measures of research productivity and impact will develop around this UK
research corpus, increasing the sensitivity and predictiveness of the
RAE analyses more and more.
(See the online impact-measuring scientometric search engines we have
developed at Southampton: http://citebase.eprints.org/cgi-bin/search
and http://opcit.eprints.org )
(20) And all that is needed for this is for RAE to revert to online
submissions, requiring online CVs linked to the full-text draft
of each paper in the researcher's institutional Eprint Archive.
http://www.eprints.org/self-faq/#research-funders-do
Assessment engines like citebase can then be used by RAE to derive
ever richer and more effective measures of research productivity
and impact, available to the RAE continuously. And institutions
could continuously monitor and improve their own research productivity
and impact, using the same measures. And the rest of the world could see
and emulate the system, and its measurable effects of research
visibility, uptake and impact.
Just a few loose ends: Books are often not give-aways, as peer-reviewed
research is, so this solution does not apply as well to the assessment
of book output -- but this was a problem even in the old system, because
impact measures are not as readily available or widely used for books.
The new system will strengthen the RAE and its accuracy and fairness in
all sectors except books. And even with books there is the option
(especially with esoteric monographs that produce virtually no royalty
revenue) to put them in the Institutional Eprint Archives too. And even
if the book itself is not accessible online, the citation of books by
the online peer-reviewed publications will be a measurable and usable
estimate of their impact.
The UK is uniquely placed to move ahead with this and lead the world,
because the RAE is already in place. But we need to move fast,
because other countries are getting the idea too: The Netherlands has
no formal RAE yet, but it is about to implement a national system of open
research archiving for all of its universities called DARE:
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/2356.html
It is just a matter of time before they realize that a marriage between
a national network of DARE-style institutional Eprint Archives and a
national RAE-style research assessment exercise makes a natural, indeed
an optimal combination. If/when they do, it will be they, and not the
UK, who lead the rest of the world toward this natural solution.
But although the naturalness and optimality -- indeed the inevitability
-- of all this is quite transparent, it is a fact that research culture
is slow to change of its own accord, even in what is in its own best
interests. But that is precisely why we have funding councils and
research assessment: To make sure that researchers do what is best for
themselves, and for research, and hence for the supporters (and
beneficiaries) of research, namely, tax-paying society: Institutional
self-archiving of research output, for the sake of maximizing research
access and impact has been much to slow in coming, even though it has
already been within reach for several years. The UK and the RAE are now
in a position to lead the world research community to the optimal and
the inevitable.
http://www.eprints.org/self-faq/#research-funders-do
We at Southampton, meanwhile, are continuing to try to do our bit to
hasten the optimal/inevitable for research and researchers. We are
planning to harvest all the metadata form the submissions to RAE 2001
http://www.hero.ac.uk/rae/submissions/ into RAEprints, a "meta-archive,"
that is meant to demonstrate what RAE returns would like if this
recommendation were adopted. Of course RAEprints (i) will contain only
four papers per researcher, rather than their full peer-reviewed research
output, (ii) it will only contain the metadata for those papers (author,
title, journal-name), not the full-text and the all-important references
cited. But we will also try to enhance the demo by adding as much of
this missing data as we can find on the Web, so as to at least give a
taste of the possibilities: Using paracite http://paracite.eprints.org/
an on-line citation-seeker that goes out and tries to find peer-reviewed
full-text papers on the web, we will "stock" RAEprints with as much as
we can find -- and then we will invite all the RAE 2001 papers to add
their full-texts too!
But we can't do it all alone. We hope HEFCE and RAE will put their full
weight behind progress toward this outcome, so beneficial to so many, in
so many ways.
Best wishes,
Stevan
-------------
Date: Fri, 25 Oct 2002 19:06:55 +0100 (BST)
From: Stevan Harnad <[log in to unmask]>
This is a response to the HEFCE "invitation to contribute"
recommendations for restructuring the RAE:
I have written two papers on how the RAE might be greatly improved in
its assessment accuracy and at the same time made far less effortful
and costly -- while (mirabile dictu) doing a great indirect service
to research and researchers, both in the UK and in the rest of the
scholarly/scientific world as well:
Harnad, S. (2001) "Research access, impact and assessment." Times Higher
Education Supplement 1487: p. 16.
http://cogprints.soton.ac.uk/documents/disk0/00/00/16/83/index.html
Harnad, S. (2001) The Self-Archiving Initiative. Nature 410: 1024-1025
http://www.nature.com/nature/debates/e-access/Articles/harnad.html
If you wish to see what the RAE would look like if UK research output
were continually accessible online, and hence continuously assessable, see:
http://citebase.eprints.org/
To see how the RAE could help hasten this outcome (which is in any case
optimal and inevitable), see:
http://www.eprints.org/self-faq/#research-funders-do
We at Southampton are currently harvesting the RAE submissions data
and putting them in an Eprint Archive to provide a "demo" of the sorts of
possibilities an online, open-access research corpus opens up for
research visibility, accessibility, uptake, usage, citation, impact and
assessability.
Best wishes,
Stevan Harnad
|
