Greater Autonomy Needed to Reform Research Reassessment: A View from Spain
The Agreement on Reforming Research Assessment offers a window of opportunity to transform evaluation systems across Europe. However, in order to implement the qualitative style of evaluation proposed, it will be necessary to modify the processes and sites of evaluation in those countries relying on centralized evaluation systems.
A sea change in European research evaluation
In July 2022 the European Commission (EC) published the Agreement on Reforming Research Assessment, a consensus document coordinated by Science Europe and the European Association of Universities and the end product of a broad consultation with over 350 organizations. This initiative is an EC response to the perception that current research assessment is not only problematic in itself, but has become a barrier to European policies, such as Open Science (OS), Integrity or Responsible Research and Innovation.
The core commitments of the agreement focus on the need for evaluation to reflect the plurality of modern research and the varying contexts within which it takes place. It states that evaluation must ‘recognize the diversity of contributions to, and careers in, research’, must be based ‘on qualitative evaluation for which peer review is central’, ‘abandon inappropriate uses (…) of journal and publication-based metrics (…)’ and ‘avoid the use of rankings of research organizations.’ In short, the agreement establishes the primacy of qualitative and peer evaluation to support appropriate contextualization and diversity of scientific practices and careers.
Last September, it was announced that a Coalition for Advancing Research Assessment would be in place by the end of 2022 with organizations willing to make a commitment to the agreement. At the time we write this, more than 125 organizations had added their signature, including major institutions such as CERN, CNRS, DFG, and the main agencies and university associations in Switzerland, Norway and the Netherlands. [As of May 12, 2023, the count stood at 546.]
Cultural, institutional and bureaucratic barriers to reform
However, the agreement and its recommendations are by no means a done deal. They require significant cultural change: it will be difficult for some researchers to stop thinking of quality in terms of journal prestige and impact factors. They will also require institutional reforms, such as the formal details that shape criteria in assessments. For example, evaluation forms that now ask for a ‘list of top ten publications’ should now shift to require the ‘most important five scientific contributions.’ However, the most formidable barriers are those processes associated with centralized and bureaucratic governance of evaluation, a critical factor in the example we explore here of Spain.
In Spain, universities can only hire and promote lecturers who have been approved by the Spanish Agency for Quality Assessment and Accreditation (ANECA) or equivalent regional agencies. This double-step recruitment process (agency ‘vetting’ before the usual job examination at the university) was implemented in the 2000s to reduce nepotism. The regular research individual incentive for lecturers (‘sexenio’) is also awarded by ANECA. This handling of individual accreditation and assessment by external agencies is rigid and standardized, which means that in practice, evaluation is dependent on journal rankings (derived from impact factors) of the publications submitted. These criteria are in flagrant conflict with international good practices (see DORA or the Leiden Manifesto), as well as with the European agreement since they create strong incentives for narrowing practices towards publication, including gaming.
Devolving autonomy to universities is the only escape from metrics
Is there any possibility of changing these criteria while maintaining a system of individual accreditation and evaluation by external agencies? Can research bureaucracies far removed from the context carry out individual evaluation that ‘recognize the diversity of contributions to, and careers’ in science?
We believe not. Bureaucratic structures require the use of standardized criteria that can be applied quickly and homogeneously to the population assessed. Such a need gives rise to procedures that are incompatible with the appreciation of diversity sought by the agreement. The pluralization and flexibility of evaluation criteria cannot be implemented from administrative machinery that must evaluate hundreds of CVs each year, and thus cannot avoid simplifying decisions using metrics. If external agencies tried to adopt contextualized and plural ex-post assessment, they would likely collapse under the time and costs required.
For Spanish universities to be able to adopt the new evaluation criteria, a change in governance is essential. In order to carry out a more plural and flexible evaluation with qualitative criteria, it is necessary to focus on the content of research and this means to evaluate fewer times, at a collective rather than individual level, and from positions closer to the practices assessed.
To achieve this, hiring and promotion decisions should be restored to universities, as is the case in most European countries. Moreover, even on its own terms, the centralized scheme has failed in its goal of preventing ‘academic in-breeding’: after 15 years of application, 73 percent of Spanish lecturers work at the same university where they read their thesis, and only 2.5 percent of public university lecturers have a foreign nationality. Yet, is it possible to have autonomy without the risk of favoritism?
Institutional accreditation: university autonomy with governmental checks
Following a proposal of the Andalusian Evaluation and Accreditation agency, ‘institutional accreditation’ of university departments or schools is one plausible model of evaluation governance that would allow a change in criteria, while preventing in-breeding. Under the scheme, universities would be given full autonomy in hiring and promotions. On the other hand, every 4 or 6 years, an external evaluation agency would certify that the centers comply with agreed criteria of quality in procedures and results. The system would allow departments to hire and promote candidates with the profiles most suited to their needs. At the same time, the accreditation agency would be able to ascertain that adequate procedures and practices are followed, so that only those centers with demonstrated capacity and results would be allowed to exercise this autonomy.
This proposal does not imply a major legislative change because institutional accreditation is a mechanism that already exists to evaluate university degrees. It would be a matter of adding components for the selection and promotion of professors to the current accreditation schemes.
Reviewing the law on Spanish research assessment
The Spanish parliament in April passed a new Law on the University System (so called LOSU). While there is agreement on the need to modernize evaluation, there is not a shared vision about how to do it. In alignment with the European agreement, the law recognizes the importance of new university missions and opens options for a more plural evaluation of their contributions. However, in contradiction to this ambition, it maintains the core elements of the existing governance with a focus on individual assessment by external agencies. Although, fortunately, the draft also introduces (but as yet does not deploy) the possibility of institutional accreditation.
Whilst we believe that external accreditation should be simply eliminated, this ambiguity in the draft leaves room for hope. Above all, the new law should aim to prevent the accreditation agencies from attempting to apply the diversifying logic of the European Agreement to bureaucratic criteria. The contradictory logics of the European Agreement and centralized assessment is unlikely to be sustainable and could lead to administrative collapse.
On the contrary, reducing the importance of individual accreditation with a simplified procedure, and developing institutional accreditation as the main filter of control, could allow for a much more efficient and contextualized research assessment system that enhances university missions to address the most pressing global challenges.
This blog post is based on an editorial letter published in the journal El Professional de la Información.
