 |
 |
Dear Stephen, Ernesto and James, Many thanks for your contribution to this discussion, which of course we always welcome. It is always fascinating to me to see the differences between the strategists and the practitioners in the responsible metrics space. I would point out that the particular question in the consultation to which we are responding asks for “comments on any points within the guidance that you feel are problematic or need amendment”. For this reason, I wonder if our response appears more negative than it actually is. We do try to make it clear that these concerns are expressed within a broader appreciation of what the Wellcome Trust is trying to achieve. Perhaps we should bolster this section to make that absolutely clear. The group that drafted the response consisted mainly of those who were DORA signatories so no negativity towards DORA is intended. However, I can see how the wording around ‘the focus on DORA and the prescriptive nature of the guidance also run the risk of turning responsible metrics into a tick-box exercise’ could be mis-interpreted. I propose removing ‘DORA and’ from this sentence. I think we all appreciate the huge contribution that DORA-the-movement is making, and seek to amplify their efforts through all of our channels. However, the Wellcome Trust guidance seeks compliance with DORA-the-Declaration, and in a way that many on the group believe will be problematic. The group felt that a broader expectation around responsible metrics (including both DORA and other approaches) would be more consistent with their original OA policy, and with enabling appropriate responsible metrics approaches within HEIs. Whilst strategists and practitioners share the same aims, those of us who actually have to operationalise these policies do view them quite differently, I think. But of course, we have until the end of the day to discuss this. We’d very much welcome thoughts from others – especially the practitioners who will end up seeking to implement this guidance in their institutions – as to whether they feel this response represents their views. Best regards Lizzie From: A bibliometrics discussion list for the Library and Research Community <[log in to unmask]> On Behalf Of James R Wilsdon Sent: 21 February 2020 13:32 To: [log in to unmask] Subject: Re: LIS-Bibliometrics response to Wellcome Trust DORA guidance consultation Dear all, With similar caveats to Stephen and Ernesto(!), I’d like to signal my support for their points. While DORA is of course not the only show in town in terms of frameworks for responsible metrics and evaluation, I think all of us who share a commitment to this agenda should recognise, welcome and support the extent to which it has become the most visible and influential articulation of a wider set of concerns. The evolution of DORA over the past 2-3 years beyond the original declaration and into a more plural, interdisciplinary platform for commitment and action on these issues has been hugely positive, and as Stephen reminds us, DORA's roadmap extends this further in a number of positive directions. I think it’s important to recognise this - and to signal collective support for DORA’s efforts - while of course preserving the space for continued innovation and diversity in the approach taken by individual organisations. This is very much the spirit in which I interpret the Wellcome guidance (and I would imagine that colleagues in Leiden would feel the same way - though am cc:ing Sarah de Rijcke, Ludo Waltman and Ismael Rafols, as they may wish to contribute!) I think one of the greatest strengths of these debates over the past 5-6 years has been the extent to which colleagues across a range of linked initiatives and organisations have worked collaboratively to strengthen and mutually reinforce each other’s efforts. Through this period, DORA has emerged as the framework with the greatest global reach and visibility, so at a pragmatic level, I think it makes tactical strength to build on this further, and continue to evolve DORA (as Stephen says) to better reflect a wide range of issues and concerns. This doesn’t mean it has to crowd out or stifle the development of other initiatives - merely that we should recognise and build on what is working well, and try to strengthen it further. best wishes, James James Wilsdon FAcSS Director, Research on Research Institute (RoRI) Digital Science Professor of Research Policy, University of Sheffield +44 (0)7710 760086; http://researchonresearch.org/ [cid:image001.png@01D5E8C0.D7EDABE0] On 21 Feb 2020, at 13:02, Ernesto Priego <[log in to unmask]<mailto:[log in to unmask]>> wrote: Dear all, Like Stephen, I contribute to this list cautiously and respectfully. As a researcher, I am keen to participate in the issues this group works on. Unlike Stephen I have no connection to DORA, but I want to thank him for sharing this response to the draft response to the consultation. It does represent my own views as an academic. I was concerned the draft was overtly critical of an initiative that I consider a huge step in the right direction. I am particularly concerned with the increased influence proprietary third-party systems have on our understanding and pracrice of research assessment, and I often wish we were as critical of the likes of Clarivate and its uptake (to name only one) as we are of researcher and funder led initiatives. I only allow to say this here in the context of Stephen's response, with which I fully agree as a mere file and rank academic author and editor. Thank you for allowing us to participate in the conversations even as somewhat external participant observers. All the best, Ernesto On Fri, 21 Feb 2020, 12:43 Curry, Stephen, <[log in to unmask]<mailto:[log in to unmask]>> wrote: Hi folks Although I’m subscribed to this email list (having participated in one of your events last year), I’m not sure that I can count myself a member of this community. But as chair of the DORA steering committee I hope it’s OK if I offer a couple of comments and some context from DORA’s perspective on the draft response. I do so cautiously and with respect for the autonomy of this community, but in a spirit of collaboration, safe in the knowledge that we have a common desire to reform research assessment. The declaration is one part of the overall DORA initiative. In my view the draft response presents a more rigid interpretation of the declaration than DORA (the organisation) has been promoting in recent years. There is indeed a focus on eliminating undue reliance on the JIF and it is true that no specific mention is made of the h-index, but the declaration<https://sfdora.org/read/> has been crafted to allow plenty of room for manoeuvre which accommodates the responsible use of metrics. For example, paragraph 6 recommends that publishers might reduce the influence of the JIF by presenting an array of other metrics to describe their journal performance; (personally, I would also like to see them present their citation distributions!). Furthermore, paragraph 15 asks researchers to focus on content “rather than publication metrics”, while paragraph 17 suggests they “use a range of article metrics and indicators on personal/supporting statements”. Both of these statements could easily be interpreted as guidance not to use simplistic aggregate indicators such as the h-index. As an organisation, DORA has been publicly critical<https://twitter.com/DORAssessment/status/1192494031869267970> of the h-index. So, it’s not quite correct to assert that “DORA doesn't seek to address the poor use of other metrics such as the h-index”. With regard to the focus on the sciences, it is indeed true that the declaration has its origins within the scientific community. However, as an organisation our present roadmap<https://sfdora.org/2018/06/27/dora-roadmap-a-two-year-strategic-plan-for-advancing-global-research-assessment-reform-at-the-institutional-national-and-funder-level/> commits us to expanding the geographical and disciplinary reach of the initiative and we have already begun working with humanities scholars to advance this agenda. I am not sure of the basis for the “concerns that both the focus on DORA and the prescriptive nature of the guidance also run the risk of turning responsible metrics into a tick-box exercise.” It is certainly the case that declarations are open to abuse by organisations that sign and only play lip-service, but again we as an organisation are focused on enabling implementation and indeed the main thrust of the Wellcome policy<https://wellcome.ac.uk/how-we-work/open-research/guidance-research-organisations-how-implement-dora-principles> is to foster real action, an emphasis that DORA the organisation shares entirely. This blog post from last year (DORA – accentuating the positive<https://sfdora.org/2019/05/09/dora-accentuating-the-positive/>) underscores how pragmatic and action-oriented we are in using the declaration to facilitate reform; it recognises that there are no one-size-fits-all solutions. We have significantly increased our activity in this area in the past couple of years, both by organising conference sessions and meetings on the practicalities of research assessment reform, and by working collaboratively with organisations (universities, funders and learned societies) that are seeking to implement DORA. These are areas where we are aiming to intensify our activities in coming years. Finally, it’s perhaps worth emphasising also that the Wellcome policy is focused not on the full text of the declaration, but on what it calls the ‘DORA principles’ (a sensible distillation of the core message that allows plenty of latitude and represents values shared with other approaches, such as the Leiden Manifesto). Consistent with this, a breadth and variety of approaches are encouraged by the Wellcome policy. It clearly states that universities are at liberty not to sign DORA if they adopt practices that embed a robust approach to research evaluation. As chair of DORA, I am entirely comfortable with that approach, though at the same time we are pleased to see how influential the declaration and the organisation have been in helping to advance this agenda. We know there are many others who share a passion for improving research assessment practice, not least the LIS-bibliometrics community (!), and we are always keen to collaborate in any way that can accelerate progress. With all good wishes, Stephen Curry On 18 Feb 2020, at 14:07, Elizabeth Gadd <[log in to unmask]<mailto:[log in to unmask]>> wrote: Dear LIS-Bibliometrics colleagues, A small group of us got together to develop a LIS-Bibliometrics response to the Wellcome Trust DORA Implementation guidance. We are now seeking the wider community’s thoughts on this to ensure it accurately reflects the majority view. The deadline for submissions is 24 February so we would welcome any thoughts you might have - on or off-list - by Friday 17 February. Best regards Lizzie LIS-Bibliometrics response to Wellcome Trust DORA Guidance Consultation 5.Please provide comments on any points within the guidance that you feel are problematic or need amendment The LIS-Bibliometrics community consists of approximately 1,000 bibliometric practitioners based mainly in Libraries, but also in Research and Planning Offices. Approximately one-third of our membership is based outside the UK. LIS-Bibliometrics welcomes the engagement of the Wellcome Trust with responsible research evaluation. Many of our members are feeding back individually or as institutions; this feedback covers points of convergence within the community. General approach The community has some concerns about extending the responsible research evaluation requirements already stated in the Wellcome Trust’s Open Access policy through a separate guidance document. It needs to be clearer how this more general DORA guidance relates to (or potentially supersedes?) the statement in the open access policy. Notwithstanding the requirement of Plan S to forge a link between better research assessment and delivering open access, it is suggested that an enhanced responsible research evaluation approach is kept mainly separate from, but linked to, the Wellcome Trust’s OA policy. The community are concerned that the focus on implementing DORA, when the Wellcome Trust OA policy is clear that HEIs may adopt any approach to responsible metrics including the Leiden Manisfesto, is unnecessarily limiting. There are many good reasons why an HEI may have chosen not to sign DORA. For example, DORA doesn't seek to address the poor use of other metrics such as the h-index and its language is inherently biased towards the sciences. We would recommend that the focus of this guidance is shifted away from DORA and towards encouraging a much broader approach to responsible research evaluation. There are concerns that both the focus on DORA and the prescriptive nature of the guidance also run the risk of turning responsible metrics into a tick-box exercise, with newcomers only doing exactly what is needed, rather than making their own commitments to responsible metrics. We would propose that a framework approach that points to areas of best practice might be better in order to allow institutions to maintain their autonomy in decision-making and directing their own paths to responsible metrics in line with their own institutional missions and values. Specific feedback on certain sections: 1) What we expect • The community have concerns that the DORA-based expectations stated in this section ultimately require institutions to sign DORA in all but name, whether it is the right thing for the institution at that time or not. It should be noted that institutional size and make-up will have a bearing on an institution’s approach to responsible metrics. It is felt that this is not catered for in the current guidance but is of great importance. • The statement “Prominently on your website” needs to be clarified. This statement needs to be made more specific or left to the judgement of HEIs as to where this information fits best on their individual websites. It was felt that it was entirely legitimate for institutions to keep some details of their approach to responsible research evaluation for internal consumption only, whilst still publicly committing to the broader principles required by the Wellcome Trust, and being willing to share the detail with the Wellcome Trust via audit. This would allow institutions to preserve institutional autonomy and develop processes and decision making as most suited to that organisation. • It is unclear whether the statement on whistle-blowing is intended to be an internal process to allow researchers to report behaviour or practice that contravenes the principles, or whether whistle-blowers are expected to go directly to the Wellcome Trust. 2) Statement of commitment • The community felt that in order to best support researchers, it was still necessary for institutions to explain what a Journal Impact Factor and h-index was, even when the institution did not use those metrics in their internal processes. It was felt that the way the guidance is currently worded, this legitimate activity may explicitly contravene the guidance provided under the statement of commitment. 3) Plan for implementing the DORA principles • Strategy & Leadership Responsible research evaluation needs to be resourced to be implemented and it is unrealistic to suppose that many HEI’s have dedicated support for this. This area would benefit from a range of examples of best practice. It needs to be clearer that the bullet points are only suggestions. • Hiring & promotion Again, this section should focus not only on the DORA principles. It would be useful to provide examples of other responsible approaches in relation to hiring. It was felt that there were good reasons to have target journals and the statement ‘discouraging the use of lists of ‘target journals’ that researchers (at all levels) should aim to publish in’ should be removed. Such lists are not always based on journal brands and metrics, but on open access status, audience, peer-review quality and turn-around times. It was felt that such lists may be business sensitive and should not be subject to the same transparency criteria as other elements of a responsible metrics policy. • Communication The scale of the training offer required was not clear and it was felt that the level of resource needed to reach the entire research community was likely to be disproportionate to the effect of that training. The community’s experience of offering such training has led them to conclude that an embedded approach, rather than a dedicated one, was often most effective. Responsible research evaluation should be part of a larger culture change. Again, it was felt that having Champions was unrealistic. This was unlikely to be a sought-after role and therefore unlikely to be an effective approach. Incorporating responsible metrics into a broader research culture change role may be more effective. • Monitoring This section is unclear as to whether institutions are required to report on their progress internally or to the Wellcome Trust. It is also unclear as to whether institutions will need to report in January 2021 or just have a plan by that date as to how they are going to report. This needs to be clarified. In particular, the wording “Should consider” is confusing. Are these reporting approaches optional or mandatory? The range of qualitative and quantitative indicators suggested is an unrealistic demand. In some institutions responsible metrics has no clear owner and it is unclear as to what benefit such a reporting regime will achieve. We recommend that this section is replaced with “HEIs should review and consider their approaches to responsible metrics”, preferably with some previously successful examples. Dr Elizabeth Gadd Research Policy Manager (Publications) Research & Enterprise Office Loughborough University Loughborough, Leicestershire, UK T: +44 (0)1509 228594 S: lizziegadd E: [log in to unmask]<mailto:[log in to unmask]> ________________________________ To unsubscribe from the LIS-BIBLIOMETRICS list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=LIS-BIBLIOMETRICS&A=1 ________________________________ To unsubscribe from the LIS-BIBLIOMETRICS list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=LIS-BIBLIOMETRICS&A=1 ________________________________ To unsubscribe from the LIS-BIBLIOMETRICS list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=LIS-BIBLIOMETRICS&A=1 ________________________________ To unsubscribe from the LIS-BIBLIOMETRICS list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=LIS-BIBLIOMETRICS&A=1 ######################################################################## To unsubscribe from the LIS-BIBLIOMETRICS list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=LIS-BIBLIOMETRICS&A=1