We were talking here yesterday here about what kind of label 5* would have, whilst acknowledging that 2* research is still pretty amazing too.
How about...
If they impose 100% return, you have to accept that will need to be 0 submissions for some people who are shiny new academics.
Sticking with the 100% return, if you are suggesting an average of 2 per FTE to keep submitted volume of outputs largely the same... why not just move to 2 outputs per individual? The only people not allowed to submit 2, are ECRs (haven't fully thought through the E&D issues here...).
Auditing portability is going to add to the burden rather than reduce it (which I think is another ambition?) and would make this proposal tricky, so let's scrap the non-portability idea too.
This then gives HEIs the space to set their own performance expectations of research quality and outputs which is less driven by REF, and more focussed on the volume and quality they believe is appropriate and achievable in their own institutions. Another Stern ambition met! It might also remove the difficulty of co-authorship - less administrative burden there too!
I'm not sure (having only just thought of the idea) if this would still lead to unpicking 4* submissions... but I think it might be a way to reduce the likelihood.. maybe? Not sure. I know we certainly didn't have everyone submitting 2x4* plus 2 others, so ... hey presto?
Jennifer
Research Office Swyddfa Ymchwil
Cardiff Business School Ysgol Busnes Caerdydd
Aberconway Building Adeilad Aberconwy
Cardiff University Prifysgol Caerdydd
Cardiff Caerdydd
CF10 3EU CF10 3EU
Phone: +44 (0)29 2087 6811 Ffôn: +44 (0)29 2087 6811
Email: [log in to unmask] Ebost: [log in to unmask]
-----Original Message-----
From: Kent, Ray [mailto:[log in to unmask]]
Sent: 27 January 2017 10:53
To: [log in to unmask]; [log in to unmask]
Subject: [Metrics] Re: Use of metrics in the REF - further thoughts
Hello again
Thanks for your replies - please keep them coming.
The difficulty I have with the REF Consultation is that according to David Sweeney, the sector doesn't have the possibility of saying 'no' to any of the Stern (2016) recommendations; through the Consultation, HEFCE is charged with implementing Stern's report, not debating its merits. Some have suggested that HEFCE is playing a double game here and really wants us to say no to some of Stern's wilder suggestions - see, for example http://wonkhe.com/blogs/stern-times-for-ref-game-players/. Whatever the case, if we do say no, it has to be the whole sector saying it (otherwise HEFCE won't be obliged to act) AND we have to offer an alternative solution, that looks Stern-like but isn't Stern.
Adding a magical 5* category - which we might term 'simply awesome' - might help reduce bunching at the top end, thereby reducing the flattening-out effect on QR. I believe this was mooted in some of the panel overview reports following REF 2014, and of course there was a 5* category in RAE 2001. What it wouldn't do, is address the decoupling proposal which is at the heart of Stern's Recommendation 2.
Do you agree with the idea of entering staff with zero or one publication, thereby allowing the main focus of REF 2021 to become the in-depth analysis of 4* papers (I am using 4* here in the sense of REF 2014)? This will necessitate the use of bibliometric indicators to distinguish the best from the rest - leading to the requirement for a 5* category, or else grade deflation as outlined in my previous posting to the list. Decoupling means that the long tail of 'weaker' papers is simply not examined. Is that a good thing? Sure, it saves the panels from looking at some less-than-stupendous research (recalling that 3* and 2* articles are still of 'international' quality); but it also makes the panels' job harder in the sense that their attention is confined to distinguishing the 'simply awesome' from the 'world-leading', i.e. splitting hairs.
Is this REALLY how we want to proceed? Is there not a better way?
Regards,
Ray
-----Original Message-----
From: Kent, Ray [mailto:[log in to unmask]]
Sent: 26 January 2017 11:26
To: [log in to unmask]; [log in to unmask] <[log in to unmask]>
Subject: [Metrics] Re: Use of metrics in the REF - further thoughts
Hello
I've been trying to think through some of the 'big questions' posed by the Consultation on REF 2021, notably what might be the outcome of implementing Lord Stern's proposal to decouple outputs from individuals (Recommendation 2 of Stern, 2016).
If Recommendation 2 is implemented, all institutions will seek to maximize their REF score by submitting only the very best (potentially 4*) outputs as judged through peer review. This would be greatly facilitated by the possibility to submit staff with no outputs, or perhaps only one output.
Faced with a submission consisting entirely, or almost entirely, of 4* outputs, how will REF panels respond? As I see it, panels will have a choice: they can either rate the whole set of outputs as 4* (applying the logic of REF 2014), or reach for bibliometric indicators in an attempt to separate 'top 4*' outputs from 'bottom 4*' outputs, with possibly a third category of 'middle 4*'.
A consequence of choosing to rate a whole set of outputs as 4*, is that in REF 2021, there could be many submissions in each UoA that consist almost entirely of 4* outputs. With Impact and Environment contributing only 35%, in total, to the weighting of QR, this would prove a real headache for Research England, HEFCE's successor, and the devolved funding councils. If the field becomes bunched-up in this way, QR will be spread much more evenly than was the case following REF 2014. One might argue that this reduction in concentration would be a good thing, but unless Research England steps in to moderate the effect on the 'Golden Triangle' and other research-intensive institutions, they would likely see a substantial drop in their QR funding. This would create significant turbulence across the sector, particularly when considered alongside the as yet largely unknown consequences of the TEF and Brexit.
Alternatively, if the REF panels decide to use bibliometrics to assign a 3* rating to 'middle 4*' and 'bottom 4*' outputs (papers that, in REF 2014, would each have received a 4* rating), the effect will be to deflate, relative to REF 2014, the grades awarded to outputs in REF 2021. If that were to happen, come 2021 we could be faced with a situation where a significant number of UK universities have the appearance of being weaker, in research terms, than they were at the time of REF 2014. That doesn't send a good message to Government when we are lobbying for more funding.
I am sure that HEFCE and its equivalent bodies in the devolved administrations have thought about this carefully, and are acutely aware of the unintended consequences that might result if the decoupling of outputs and individuals were to be implemented. However, if as a sector we fail to strongly object to Recommendation 2 in our Consultation responses (the temptation being to focus on less complex issues), the funding councils will have no grounds for rejecting the proposal.
Let's make our voices heard, and say no to Recommendation 2!
When doing so, we should propose an alternative way forward. This could be to retain the approach to outputs taken in REF 2014, or there may be a better way - ideally one that doesn't involve having to submit Individual Staff Circumstances. Any thoughts?
Best regards,
Ray
--
Dr Ray Kent
Director of Research Administration
Research Office
The Royal Veterinary College
Royal College Street
London
NW1 0TU
T: +44 (0)20 7468 1206
E: [log in to unmask]
[RVC Logo - link to RVC Website]<http://www.rvc.ac.uk> [Twitter icon - link to RVC (Official) Twitter] <http://twitter.com/RoyalVetCollege> [Facebook icon - link to RVC (Official) Facebook] <http://www.facebook.com/theRVC> [YouTube icon - link to RVC YouTube] <http://www.youtube.com/user/RoyalVetsLondon?feature=mhee> [Pinterest icon - link to RVC Pinterest] <http://pinterest.com/royalvetcollege/> [Instagram icon - link to RVC Instagram] <http://instagram.com/royalvetcollege>
This message, together with any attachments, is intended for the stated addressee(s) only and may contain privileged or confidential information. Any views or opinions presented are solely those of the author and do not necessarily represent those of the Royal Veterinary College (RVC). If you are not the intended recipient, please notify the sender and be advised that you have received this message in error and that any use, dissemination, forwarding, printing, or copying is strictly prohibited. Unless stated expressly in this email, this email does not create, form part of, or vary any contractual or unilateral obligation. Email communication cannot be guaranteed to be secure or error free as information could be intercepted, corrupted, amended, lost, destroyed, incomplete or contain viruses. Therefore, we do not accept liability for any such matters or their consequences. Communication with us by email will be taken as acceptance of the risks inherent in doing so.
|