Dear Suan

Honestly, my work is futile, but you'd like to see it! Well, you shall see
it then, but I think I'd better tidy it up a bit before exposing it to your
stern examination.

I don't think we can blame anyone for comparing the results to last time.
More goals were scored this season. Were the goal posts wider? Or were the
teams better at scoring? Hard to tell, but if Celtic came bottom of the
league, we wouldn't ignore it even if there were new rules, we'd still
compare the outcomes. Just as the funding councils had to compare outcomes.

Let me offer you a couple of examples. One of the things my analysis
indicates is that the proportion of research rated at an international level
of excellence in 2001 was c27%. It was 55% in 2008. The proportion rated
sub-national seems to have been c10% and now it is 1.5%. No change was made
in the definition of international, but do the comparisons suggest some
change in the de facto definition? If not, it's a rate of progress that the
sector can be proud of. I don't believe it's in the sector's interests to
refrain from proclaiming an improvement because the rules have changed.

Another - I'm interested in the weightings used. How much more is a 4* piece
of work worth than a 3* piece of work? I don't mean what funding formula has
HEFCE employed to make its allocations acceptable to the minister. I mean
how much more is it worth from first principles? The VC of Leeds noted in
last week's THE that his university has risen to 14th in the league tables.
But that's on a 4:3:2:1 valuation, which seems to give too little
selectivity. On the likely funding formula weighting (which arguably gives
too much), Leeds are 24th. While the University of the Arts (42nd on the
4:3:2:1 basis) are 12th. That may or may not be right, but those weightings
will be applied because HEFCE has to compare with 2001 - maintain/increase
the extreme level of selectivity already in its formula and ensure that the
outcome for the top three approximates to their 2001 outcome. We need to try
to understand those comparisons.

I do agree that no comparison can be perfect, but I don't think that means
there should be no attempt. I'm sure somebody must be doing work on
comparisons, and that it's far better than my humble efforts.

Best wishes, and thanks for replying

Anthony 

-----Original Message-----
From: Academic, financial or space planning in UK universities
[mailto:[log in to unmask]] On Behalf Of Susan RothwellSmith
Sent: 20 January 2009 09:06
To: [log in to unmask]
Subject: Re: Analysis of RAE 2008

Dear Anthony,

These are my personal musings and do not represent the views of Dundee or
the RAE:

Isn't this retro-engineering precisely what we are warning the newspapers
against. The methodology and submission practices of the RAE2008 were so
different from 2001 that no quantitive comparison can be anything but wholly
dependent on the variables applied and the findings are therefore somewhat
suspect.   As an example, in 2001 it was intimated to the Scottish HEIs that
there would be no funding advantage to having 5* rather than 5 and this is
widely believed to have influenced the submissions of these institutions.
Comparisons also fail to recognise the contributions of outputs, environment
and esteem to the 2008 score and the variations of those contributions
between each panel.

Theme B suggests that there is a normal GPA which most panels should be
close to (and gives a credibility to the idea of a GPA of which I am very
much unconvinced in the first place) - but we are all aware that some
disciplines are stronger than others.  The RAE2008 was  largely a peer
review exercise - there was oversight within cognate groups and across the
exercise as well as from international experts.  Whilst we might look
askance at some of the outcomes are we really in a position to tell these
experts that they got it wrong?


None of which means I won't be studying your findings with great fascination
if they're ever available!

Best wishes

Susan

Susan Rothwell Smith
Planning Officer
University of Dundee
Nethergate
Dundee
DD1 4HN

The University of Dundee is a Scottish Registered Charity, No. SC015096.


01382 385441
>>> ayoungster <[log in to unmask]> 01/19/09 12:49 PM >>>
Dear all

 

I was Director of Planning at Sussex until a few years ago, and I'm grateful
to Alison Hartrey for allowing me to continue sharing these pages with you.
My reason for writing is that I've been unable to resist doing some amateur
analysis of the results of the RAE, and I wanted to ask the forum whether
anyone who has done anything similar would like to exchange thoughts.

 

Two main themes:

 

a. comparing the 2008 results to 2001 - re-expressing the definitions of
each of the old grades in terms of average proportions of the new star
levels, and using that matrix to convert the 2001 results to 2008 format.
Then comparing them using various alternative weightings of the star levels
(and the weighting is quite an interesting discussion in itself). Being
based on averages, this method can't give very reliable indications at
individual institution level, but I would argue it is unlikely to be far out
when applied to aggregations. 

 

The key conclusions from it are that the top three do not appear to have
progressed at anything like the same rate as their closest competitors since
2001, and of course that the new universities have done very much better
than before.

 

The new funding formula itself isn't something in which I'm particularly
interested, but this approach does provide fairly clear pointers to what it
will be in order to protect the top few institutions. 

 

b. under and over-rating by panels - I have attempted a vaguely systematic
assessment of the extent to which individual panels under-rated or
over-rated. This compares the weighted average of each panel's GPAs with the
weighted average of the overall GPA of each of the HEIs presenting to that
panel (weighted by the HEI's volume in that panel). One can also use
judgement to assess whether there is any good reason for the GPA for that
discipline to be higher or lower than the average for the HEIs in it - e.g.,
it may be credible that a professional/vocational area might be slightly
less research-intensive than average, and thus have lower grades. 

 

This analysis reveals considerable variations and suggests what adjustments
would be appropriate. In most cases these will broadly even out at
institutional level, but some individual departments will have justifiable
complaints if GPA comparisons are made within a given institution without
appropriate adjustment.

 

If any of this is of any interest, I'd be most grateful for your views, or
indeed for you to forward it to anybody else who might be thinking about
these issues.

 

Many thanks

 

Anthony Young


The University of Dundee is a registered Scottish charity, No: SC015096