A really good point Ian. As a community we should lobby for this.
Lizzie
From: A bibliometrics
discussion list for the Library and Research Community [mailto:[log in to unmask]]
On Behalf Of Rowlands, Ian
Sent: 04 January 2017 13:54
To: [log in to unmask]
Subject: Re: Metrics in the next REF
An excellent thoughtful response as always, Lizzie.
I’m concerned, as you are, about the leverage that even a small number of outliers or extreme values can exert on the top line figure, especially as so many indicators are still based on the arithmetic mean …
We need to wise up to this as a community (I include the data suppliers in this invitation!) and put measures around that uncertainty. Leiden do this in their rankings using stability intervals.
I’ll quote from their web site:
Stability intervals
Stability intervals provide some insight into the uncertainty in bibliometric statistics. A stability interval indicates a range of values of an indicator
that are likely to be observed when the underlying set of publications changes. For instance, the PP(top 10%) indicator may be equal to 15.3% for a particular university, with a stability interval ranging from 14.1% to 16.5%. This means that the PP(top 10%)
indicator equals 15.3% for this university, but that changes in the set of publications of the university may relatively easily lead to PP(top 10%) values in the range from 14.1% to 16.5%. The Leiden Ranking employs 95% stability intervals constructed using
a statistical technique known as bootstrapping.
(From
http://www.leidenranking.com/information/indicators)
These intervals are clearly very important in terms of framing an interpretation, especially so because they get wider as the sample size shrinks. How may wrong-headed decisions have been made on the basis that
14.5 is evidently smaller than 16.5? Define the uncertainty and we’ve done our best.
Ian
From: A bibliometrics discussion list for the Library and Research Community [mailto:[log in to unmask]]
On Behalf Of Elizabeth Gadd
Sent: 04 January 2017 13:28
To: [log in to unmask]
Subject: Re: Metrics in the next REF
Dear Katie,
My thoughts are as follows:
Do you agree with the proposal for using quantitative data to inform the assessment of outputs, where considered appropriate for the discipline?
Yes, the use of quantitative data can support (but not supplant) the assessment of outputs in some disciplines and there is anecdotal evidence that this is already happening informally, for example to aid decisions in borderline cases.
Any use of bibliometric data should adhere to the Leiden Principles for responsible research assessment, namely, take into account disciplinary differences in publication and citation practices, use a ‘basket’ of metrics
to avoid gaming; and avoid false precision (the error bars on citation data can be large). The data and calculations used should also be open and transparent to allow those being evaluated to verify the analysis.
If you agree, have you any suggestions for data that could be provided to the panels at output and aggregate level?
The challenge of using citation data to assess individual outputs submitted to REF is that many of them will be recent and may not have received many or any citations. Early citation will depend on the publication speeds within
the discipline, but can be somewhat serendipitous. It would certainly not be sensible to use any form of citation-based indicator for 6 months after publication, and the first 18 months-worth of data can be ‘lumpy’. The use of indicators that normalise by
year will ameliorate this to an extent. It would be important to exclude self-citations from any calculations to avoid gaming.
The Field-Weighted Citation Impact indicator provided by Elsevier, as suggested by the REF consultation, does normalise by year (and ASJC subject category and publication type). However, it is not helpful for the assessment of individual
papers as it looks at actual vs expected citation rates which can be too easily skewed by small numbers of ‘unexpected’ citations. The FWCI is better used for larger publication sets. The use of field-weighted citation percentiles is preferable as the “performance”
of papers is scaled from 1-100, although again, the percentile can be skewed if there are large numbers of uncited items in the dataset. It would be important to be aware of the impact of the citation performance of different publication types when using
these indicators: a conference paper has to achieve fewer citations than a journal paper to reach the top percentiles, but this does not necessarily mean that the conference paper is of higher quality.
At aggregate level, the FWCI (or equivalent) may be more useful, but again, can be skewed by a small number of high-performing papers. Percentage outputs in top field-weighted citation percentiles would be a better measure, coupled
with other indicators such as percentage papers cited at all (excluding self-cites). An issue here is whether the analysis is performed only on the submitted outputs, or on the entire output from the institution during the census period. The latter would
provide a more accurate picture of the institution’s performance within the discipline (although defining disciplines is another challenge) but it would be hard to disaggregate those papers written by staff who are not technically REF-submissible.
To overcome the problem of recent publications not receiving many citations, it might be tempting to look at journal-based citation indicators (SNIP, SJR, etc) as evidence of the paper’s quality. Whilst there is evidence of a relationship
between the citedness of journals the citedness of papers within those journals in some cases, it would be a mistake to use journal citedness as a proxy for quality for REF assessment purposes. This should be avoided at all costs.
In addition to citation indicators, another data element that should be considered by the panels is the number of authors on the paper. Hyper-authored papers are on the increase and raise questions about the contribution of individuals
and their institutions to those outputs. Providing a % contribution at an institutional level might be helpful to panels.
If quantitative data is to be used to inform the assessment of outputs, all panel members doing the assessments would need training to ensure they understand the meaning of the indicators, how they should be used and what level of confidence
they can place in them.
Hope that’s helpful and chimes with others’ thoughts!
Best regards,
Elizabeth
Elizabeth Gadd MSc, MCLIP, FHEA
Research Policy Manager (Publications)
Research Office
Loughborough University
Loughborough, Leicestershire, LE11 3TU
Tel: 01509 228594
Skype: lizziegadd
Twitter: @lizziegadd
Working Hours:
Mon 8.30-5
Tues 8.30-3
Wed 8.30-3
Google
Scholar Citation Profile
http://orcid.org/0000-0003-4509-7785
http://about.me/elizabeth.gadd
From: A bibliometrics
discussion list for the Library and Research Community [mailto:[log in to unmask]]
On Behalf Of Katie Evans
Sent: 04 January 2017 09:02
To: [log in to unmask]
Subject: Metrics in the next REF
Dear All,
The four UK higher education funding bodies are seeking views on their proposals for the next Research Excellence Framework, see:
http://www.hefce.ac.uk/pubs/year/2016/201636/
The Lis-Bibliometric Committee would like to submit a response on behalf of the List, specifically to question 18:
“Do you agree with the proposal for using quantitative data to inform the assessment of outputs, where considered appropriate for the discipline? If you agree, have you any suggestions for data that could be provided to the panels at output
and aggregate level?”
See paragraphs 73 & 74 of the consultation document (p17).
Please send your thoughts on this to the List and/or me ([log in to unmask]) by 6th Feb.
This is most relevant to those of you in the UK, but contributions from colleagues outside the UK are also welcome.
Kind regards,
Katie (on behalf of the Lis-Bibliometrics Committee)
--
Katie Evans, MMath, MSc Econ, MCLIP
Research Analytics Librarian
University of Bath
Tel: 01225 384488
Working pattern: Mon - Thurs