Bibliometrics & Responsible Research Evaluation: Responsible Metrics & DORA
Using Research Metrics Responsibly
Whilst bibliometric indicators can be a useful source of information when used responsibly, they are very narrow and limited measures and should not be used in isolation. Combining a wide range or "basket" of contextualised metrics across a range of appropriate dimensions can help to supplement or inform expert, qualitative peer review and assessment. A broad range of outputs and activities should be recognised as part of any research evaluation process, not just publications. This may include aspects such as supervision and mentorship, openness, peer review and editorial roles, contribution to economy, society and policy, awards & honours, public engagement activities etc.
The Statement on the Responsible Use of Research Metrics in UCD identifies key principles that guide the use of research metrics including:
- Fair, transparent and robust research assessment procedures
- The criteria used to evaluate research performance will be explicit
- Research metrics will be used to support, but not supplant, qualitative expert assessment.
- Journal Impact Factors will not be used as a surrogate measure of the quality of individual research articles.
- We will take into account the many different dimensions of research, and also the variation across disciplines.
- The value of all relevant research outputs (such as datasets, software and creative works as well as publications) and other types of contributions, such as training early-career researchers and influencing policy and practice, will be recognised in assessment
- UCD's commitment to equality, diversity and inclusion will be evident in how we use research metrics, recognising that many factors may impact upon an individual’s research performance and/or research metrics, including career stage, career breaks, statutory leaves and part-time working, and reflecting the UCD Equality, Diversity and Inclusion Policy and the University’s ten equality grounds.
- UCD Research: How to Write a Narrative CV (intranet)Use your UCD Connect log-in to access
- Using SciVal responsibly: a guide to interpretation and good practiceGuide by Dr Ian Rowlands, the LIS-Bibliometrics Committee, and Dr Elizabeth Gadd (2020)
- Responsible research evaluation starts hereJonathan Adams from Clarivate Analytics discusses how to responsibly use and interpret bibliometric data through a list of caveats and requirements prior to starting the planning of research evaluations.
San Francisco Declaration on Research Assessment
- San Francisco Declaration on Research Assessment - DORAThe San Francisco Declaration on Research Assessment (DORA, 2012) is a worldwide initiative covering all scholarly disciplines. The signatories are concerned about the increasing use of the Journal Impact Factor as a tool to evaluate and compare the research output of individuals and institutions.
Initiated in 2012, the American Society for Cell Biology and a group of editors and publishers of scholarly journals drafted and circulated a declaration that recognises the need to improve the way in which the outputs of scientific research are evaluated.
"The Journal Impact Factor, as calculated by Thomson Reuters, was originally created as a tool to help librarians identify journals to purchase, not as a measure of the scientific quality of research in an article. With that in mind, it is critical to understand that the Journal Impact Factor has a number of well-documented deficiencies as a tool for research assessment. These limitations include:
- citation distributions within journals are highly skewed
- the properties of the Journal Impact Factor are field-specific: it is a composite of multiple, highly diverse article types, including primary research papers and reviews
- Journal Impact Factors can be manipulated (or "gamed") by editorial policy
- data used to calculate the Journal Impact Factors are neither transparent nor openly available to the public."
The Leiden Manifesto for Research Metrics
"To support researchers and managers, five experts led by Diana Hicks, professor in the School of Public Policy at Georgia Institute of Technology, and Paul Wouters, director of CWTS at Leiden University, have proposed 10 principles for the measurement of research performance: the Leiden Manifesto for Research Metrics published as a comment in Nature." - April 23rd 2015.
CoARA - Coalition for Advancing Research Assessment
- CoARA - Coalition for Advancing Research AssessmentThe CoARA Agreement on Research Assessment Reform (2022) was drawn up in order to maximise the quality and impact of research. It recognises the diverse outputs, practices and activities of research, researchers and research organisations. This is a key movement in research assessment, marking a step towards more inclusive research assessment based primarily on qualitative judgement supported by responsible use of quantitative indicators. University College Dublin is a signatory of this agreement and member of the CoARA coalition.
Research Culture at the University of Glasgow
Funding Agencies & Responsible Metrics
Many funding agencies are now DORA signatories, including Science Foundation Ireland and the HRB.
In 2021 Wellcome Trust introduced a new open access policy that requires Wellcome-funded organisations to publicly commit to the principle of evaluating research based on the intrinsic value of the work rather than the venue it is published in: "For example, they can sign the San Francisco Declaration on Research Assessment, Leiden Manifesto or equivalent. We may ask organisations to show that they’re complying with this as part of our organisation audits."
Other Useful Resources on Responsible Research Metrics
- UK Progress towards the use of metrics responsiblyA report from the UK Forum for Responsible Research Metrics - 3 years on from the Metric Tide
- DORA, the Leiden Manifesto & a university’s right to chooseBlog post by Dr Elizabeth Gadd (Research Policy Manager) outlining Loughborough University's approach to ensuring research metrics and used responsibly.
- Responsible metrics: what's the state of the art?Presentation by Dr Elizabeth Gadd (Loughborough University) given at the FORCE2019 conference in Edinburgh on Wednesday 16 October, 2019
- How a working group began the process of DORA implementation at Imperial College LondonBlog post by Prof. Stephen Curry (Imperial College London and DORA)
Examples of Research Evaluation Statements / Positions
- INORMS Research Evaluation Working Group ResourcesA range of resources including a five-stage process for evaluating responsibly, called ‘SCOPE’
- Good practice in researcher evaluation.Recommendation for the responsible evaluation of a researcher in Finland, published March 2020
- Principles for the use of indicators in research assessment and managementUniversity of St. Andrews, November 2019
- Responsible use of research metricsLoughborough University, November 2019
- Room for everyone's talentPosition paper from the VSNU, NFU, KNAW, NWO and ZonMw (the Netherlands), November 2019
- Strategy Evaluation Protocol 2021-2027 (Netherlands)The Strategy Evaluation Protocol (SEP) describes the aims and methods used in order to assess research at Dutch universities as well as at NWO and KNAW institutes every six years (March 2020)
- Statement on the Responsible Use of Research MetricsUniversity of Edinburgh, April 2019