Bibliometrics & Responsible Research Evaluation: Responsible Metrics & DORA
Using Research Metrics Responsibly
Whilst bibliometric indicators can be a useful source of information when used responsibly, they are very narrow and limited measures and should not be used in isolation. Combining a wide range or "basket" of contextualised metrics across a range of appropriate dimensions can help to supplement or inform expert, qualitative peer review and assessment. A broad range of outputs and activities should be recognised as part of any research evaluation process, not just publications. This may include aspects such as supervision and mentorship, openness, peer review and editorial roles, contribution to economy, society and policy, awards & honours, public engagement activities etc.
The Statement on the Responsible Use of Research Metrics in UCD identifies key principles that guide the use of research metrics including:
- Fair, transparent and robust research assessment procedures
- The criteria used to evaluate research performance will be explicit
- Research metrics will be used to support, but not supplant, qualitative expert assessment.
- Journal Impact Factors will not be used as a surrogate measure of the quality of individual research articles.
- We will take into account the many different dimensions of research, and also the variation across disciplines.
- The value of all relevant research outputs (such as datasets, software and creative works as well as publications) and other types of contributions, such as training early-career researchers and influencing policy and practice, will be recognised in assessment
- UCD's commitment to equality, diversity and inclusion will be evident in how we use research metrics, recognising that many factors may impact upon an individual’s research performance and/or research metrics, including career stage, career breaks, statutory leaves and part-time working, and reflecting the UCD Equality, Diversity and Inclusion Policy and the University’s ten equality grounds.
San Francisco Declaration on Research Assessment
Initiated in 2012, the American Society for Cell Biology and a group of editors and publishers of scholarly journals drafted and circulated a declaration that recognises the need to improve the way in which the outputs of scientific research are evaluated.
"The Journal Impact Factor, as calculated by Thomson Reuters, was originally created as a tool to help librarians identify journals to purchase, not as a measure of the scientific quality of research in an article. With that in mind, it is critical to understand that the Journal Impact Factor has a number of well-documented deficiencies as a tool for research assessment. These limitations include:
- citation distributions within journals are highly skewed
- the properties of the Journal Impact Factor are field-specific: it is a composite of multiple, highly diverse article types, including primary research papers and reviews
- Journal Impact Factors can be manipulated (or "gamed") by editorial policy
- data used to calculate the Journal Impact Factors are neither transparent nor openly available to the public."
The Leiden Manifesto for Research Metrics
"To support researchers and managers, five experts led by Diana Hicks, professor in the School of Public Policy at Georgia Institute of Technology, and Paul Wouters, director of CWTS at Leiden University, have proposed 10 principles for the measurement of research performance: the Leiden Manifesto for Research Metrics published as a comment in Nature." - April 23rd 2015.
Research Culture at the University of Glasgow
Funding Agencies & Responsible Metrics
Many funding agencies are now DORA signatories, including Science Foundation Ireland and the HRB.
In 2021 Wellcome Trust introduced a new open access policy that requires Wellcome-funded organisations to publicly commit to the principle of evaluating research based on the intrinsic value of the work rather than the venue it is published in: "For example, they can sign the San Francisco Declaration on Research Assessment, Leiden Manifesto or equivalent. We may ask organisations to show that they’re complying with this as part of our organisation audits."