RAE panels DO consider anything and everything; it's not just rhetoric. 
It's  not true that RAE panellists rely on the track record of the journal 
that the material appeared in, and I'm glad that this is indeed the way the 
RAE works because journal referees can and do sometimes make mistakes when 
they accept an article for publication.  Panels will  of course have 
different policies re what proportion of material they will read in detail, 
but I would be surprised if a major criterion was "this appeared in Nature 
so we don't need to read it".

My research has shown a **strong correlation** between citation counts **of 
individuals in a department** and RAE score for department that they work 
in, but has shown **no correlation whatsoever** between **impact factor of 
journals** submitted and RAE score obtained by department. Departmental 
heads should NOT rely on journals with high impact factors to get a good RAE 
score.

So whilst I agree with Stevan that preferably refereed material should be 
submitted, that doesn't mean one should  refuse to submit non-refereed 
material or material from a low impact factor journal.

Charles

Professor Charles Oppenheim
Department of Information Science
Loughborough University
Loughborough
Leics LE11 3TU

Tel 01509-223065
Fax 01509-223053
e mail [log in to unmask]
----- Original Message ----- 
From: "Stevan Harnad" <[log in to unmask]>
To: <[log in to unmask]>
Sent: Monday, March 20, 2006 12:19 PM
Subject: Re: RAE and IRs


> Dear Rachel, I am afraid we are going around in circles:
>
> It is uncontested that RAE formally states:
>
>    (1) You can submit anything
>    (2) We will evaluate everything
>    (3) We don't count citations
>
> But the fact is that (1) is untrue, because your department will not
> (sensibly, mercifully) let you submit just anything. It won't even
> submit you as an active researcher if you can't come up with 4 items that
> meet their criteria (which are, in many fields, sensibly: articles in
> high-impact refereed journals and conference proceedings).
>
> And the fact is that (2) is not only untrue, but absurd, because RAE
> has neither the time nor the expertise to (re)-evaluate everything;
> hence the only sensible thing to do (and they do it) is for the RAE
> evaluators to rely on the track-record of the journal that peer-reviewed
> the submitted article.
>
> And the fact is that although it is true that RAE do not count citations
> explicitly (3), both the (sensible and realistic) departmental stress
> on high-impact peer-reviewed journal submissions (1) and the necessary
> (and sensible) taking-into-account by the RAE evaluators of the track
> record of the peer-reviewed journal in which an article appeared (2)
> (rather than any absurd and quixotic attempt to re-do peer review in-house
> for all of the UK's research output!) result in the many times reported
> outcome that the RAE rankings are highly correlated with (hence highly
> predictable from) citation rankings.
>
>    http://www.ariadne.ac.uk/issue35/harnad/
>
> I have several times promised to bow out of this discussion, but I do find
> it mind-boggling how it manages to keep getting bogged down in matters of
> superficial form while completely missing matters of substantive content!
>
> I've gone on to re-annotate the water that has already gone under the
> bridge in this meandering discussion (below) to try to spare some of
> us from just repeating history:
>
> On Mon, 20 Mar 2006, Rachel Bruce wrote:
>
>> Going back to the discussion on RAE rules re publication I have spoken to 
>> a
>> colleague at HEFCE to get their view:
>>
>> RAE Guidance on submissions (RAE 03.2005) allows any form of publicly
>> available assessable output to be submitted as well as confidential 
>> reports
>> which may be submitted if the HEI has prior permission that the outputs 
>> may
>> be made available for assessment.
>
> May be made available. But will they get you a good rank? And will your
> department allow you to submit just anything (at the risk of its rank)?
>
>> A proportion of outputs submitted to the
>> RAE are not text-based, and hence the RAE refers to outputs rather than
>> publications.
>
> Irrelevant to the matter under discussion.
>
>> In the generic statement on criteria and working methods (at the 
>> beginning
>> of each set of subject based panel criteria), it is clearly stated that 
>> "all
>> forms of research output will be assessed on a fair and equal basis.
>
> A pious platitude, with no substance, one way or the other.
>
>> Sub-panels will neither rank outputs, nor regard any particular form of
>> output as of greater of lesser quality than another per se.
>
> Utter, abject nonsense, immediately contradicted by the next sentence.
>
>> Some panels may
>> specify in their criteria that where they do not examine an output in 
>> detail
>> , they may use, as one measure of quality, evidence that the output has
>> already been reviewed or refereed by experts (who may include users of
>> research) and has been judged to embody research of high quality.
>
> An otiose and long-winded way of saying "please ignore what I just
> said in the previous sentence."
>
>> no panel
>> will use journal impact factors as a proxy measure for assessing 
>> quality."
>> (paragraph 32 of the Generic statement (RAE 01.2006)).
>
> "And just in case that has not confused you sufficiently, I shall now
> contradict it again."
>
> (In taking the journal's name and quality track-record into account,
> you are implicitly influenced by its impact factor, whether or not you
> do an explicit calculation -- which the RAE does not do. [Btw, if it had
> any sense, the RAE *would* count submitted research/researcher citations
> explicitly -- not journal impact factors, which are blunt instruments,
> being merely the average for the journal: That way RAE would realise how
> much it is already relying on citation impact implicitly, and it might
> at also scrap a lot of useless make-work and time-wasting on the part of
> both the submitters and the evaluators!)
>
>> From: Morag Greig, 16 March 2006 12:43
>>
>> At Glasgow we found that particularly in the initial stages of setting up
>> our repository some academics were under the impression that we were 
>> asking
>> them to stop publishing in peer reviewed journals and to simply make 
>> their
>> research available in the repository.
>
> Solution: Correct the impression: They are being asked to make their
> published, peer-reviewed research available in the repository.
>
>> This lead to concerns that such a move
>> would have an adverse impact on performance in the RAE, as there seemed 
>> to
>> be a fairly widespread belief that the RAE panels would consider such
>> 'publications' of low value.
>
> The widespread belief is correct, but since no one is proposing
> to submit unpublished work to RAE, but simply to deposit published work
> in the IR (for submission to the RAE), nothing is at issue.
>
>> The issue of making publications being submitted to the RAE available in 
>> a
>> repository is different, as here I assume you would be talking about
>> material that had been published in a peer-reviewed journal, conference
>> proceeding etc..
>
> Both published and unpublished work can be deposited in an IR, but
> published, peer-reviewed journal articles are the primary, high-priority
> target.
>
> Submitting to the RAE via IRs is a separate matter, but there too,
> published, peer-reviewed journal articles are the primary target.
>
>> From: John Murtagh, 16 March 2006 10:32
>>
>> Interesting debate on the Research Assessment Exercise and IRs.
>> From the responses so far I tend to agree with John Smith at Kent that
>> unpublished material is (or will in the future) be as much worth to
>> research output for an institution as anything else, as long as it is
>> used and cited. That this shouldn't necessarily be a so-called vanity
>> project.
>
> This is an unrealistic fallacy. Vide supra.
>
>> "...once we move away from the concept of 'impact factors' and use
>> individual article citation analysis instead,
>
> So far, so good, since individual citation counts are a sharper
> instrument than journal averages (though the journal deserves a
> separate component in the regression equation too).
>
>>  peer review will merely
>> be the first stage (and not necessarily the most important) in assessing
>> the real value of someone's work."
>
> Nonsense in the case of most work -- though no doubt some values (or
> errors) will come out in the post-peer-review wash too, in some cases,
> individual peer judgement, be it ever so expert, being fallible too...
>
>> I was interested to know from Morag Greig's response about how the RAE
>> factored into the "confusion" of academics who were unwilling to
>> deposit into the IR because of it not being a substitute for
>> peer-review.
>
> Please, let us now dispel this confusion, rather than compounding it.
>
>> Could Morag describe this a little more? I would like to use the RAE
>> exercise as an inducement, but it sounds like a deterrent from
>> depositing (in the gathering of materials at least).
>
> RAE is an inducement to depositing in an IR: It's not only a more
> efficient way to submit to RAE, but self-archiving (early!) raises
> research impact, hence RAE rating.
>
>    http://opcit.eprints.org/oacitation-biblio.html
>
>> From: Morag Greig, Wed, 15 Mar 2006
>> Subject: Re: Generic Rationale and Model for University Open Access
>> Mandate
>>
>> It's worth pointing out that many repositories have gone to some
>> lengths to explain to authors that repositories are (in most cases) NOT a
>> substitute for publication in a peer reviewed journal (whether that be an 
>> OA
>> journal or a subscription journal). Confusion over this issue has lead to 
>> many
>> academics being unwilling to deposit in their institutional repositories.
>
> Hear, hear!
>
>> From:    Leslie Carr
>>
>> Whether articles are disseminated by an institutional repository is
>> irrelevant to RAE assessment.
>>
>> It is whether they have been published by a journal (or conference or
>> whatever the assessment panel uses as a criterion) that matters.
>>
>> In other words, appearing in an IR does not make an article
>> "published" in the eyes of the RAE.
>
> Spot-on (for those with eyes to see and ears to hear!).
>
>> On 15 Mar 2006, at 09:50, Wolfgang Greller wrote:
>>
>> Can anyone tell me whether articles published in an institutional e-
>> print repository count as publication in UK RAE terms?
>
> No it doesn't.
>
>> From:    "C.Oppenheim"
>>
>>  I don't agree with Les.
>>
>> Panels are instructed to consider *any* sort of published output, and
>> this would consider something that had only ever appeared on an IR, or
>> on a web page; indeed, statistics from the last RAE show that a small
>> proportion of the items examined by the panels were in web page form
>> only.  However, if you want to impress the RAE panel, the item should
>> have been through a proper refereeing process.
>
> Charles later conceded that he was speaking about RAE's formal criteria,
> not actual submission practice, let alone successful/optimal practice
> (in terms of RAE outcome), by departments.
>
>> From: Stevan Harnad
>>
>> All this means is that RAE also considers unpublished papers. It has
>> nothing whatsoever to do with whether or not those papers were self-
>> archived on the web. (Plus, as we know, and as many studies by
>> Charles himself has shown, the RAE outcome is highly correlated with
>> the citation counts of the journal articles submitted; and
>> Departments place high -- too high -- emphasis on the "citation
>> impact factor" of the journal in which the 4 articles selected for
>> submission were published.)
>>
>> So although the RAE nominally considers all papers, and it definitely
>>
>> does not count citations directly, there is considerable emphasis on
>> journal articles on both sides. Moreover, I profoundly doubt that
>> many RAE submissions actually get read (let alone read by someone as
>> competent to evaluate them as the referees of the journal in which
>> the originally appeared). So the fate of an unpublished manuscript
>> amounts to the fate of  a book judged by its cover (without even the
>> help of its publisher and his prior track record for quality)...
>
> Amen
>
> Stevan Harnad
>