The Stern Review of the REF – An Economist Against Markets!

Feel free to go anywhere you want – as long as you stay here.

Historically, UK public universities have received two main funding streams from government – T for teaching and R for research. Although these are calculated by reference to subject mix and other weighting factors, they arrive as a block grant for university managements to distribute according to their own strategic decision. The T stream has been largely replaced by student fees but the R stream continues. This currently amounts to about £2 billion per year. Its largest component – QR – is distributed according to judgments of the aggregate quality of research across the whole university.

Since 1986, the calculation of QR has been based on a national quality review, currently known as the Research Excellence Framework (REF). Currently, panels of subject experts evaluate organizational units in a discipline (usually departments) by reading selected publications, assessing their support for research and examining the wider social or economic impact of their work. The process has been through seven iterations. After each, there have been complaints about the costs, the administrative burdens and the gaming of the rules by allegedly less scrupulous institutions.

Most cycles have, then, concluded with a review intended to correct the deficiencies of the previous round. REF 2014 was no exception and the review, by a panel chaired by Lord Stern, an eminent economist, has just been published.

The review underlines three important points. The first is the inevitability of audit. If public universities are receiving public money to spend at their own discretion on research, then they must expect to account for the use of that money and its relevance, in the broadest sense, to the concerns of taxpayers. Second, the exercise is not unduly costly: £200 million to allocate £2 billion per year for seven years until the next REF is less than 1 per cent overhead. Third, the emphasis is on assessing units rather than individuals. Critics often overlook this. However, when I was an assessor in 2008, we were firmly instructed to destroy all data relating to individuals at the earliest opportunity, in case of a freedom of information request. The only output that mattered was the collectively-agreed overall set of scores for the unit. This was all that was retained.

Lord Stern’s committee also disposes of the idea that peer review of publications should be replaced by bibliometrics. This had been canvassed as a way to reduce the overhead costs of the exercise. Although strongly favored in some policy circles, the idea had been seriously battered by an earlier report, chaired by James Willsdon, a leading science policy analyst. It is hard to see it being revived, although the idea has merits as a check on some of the potential for corruption in peer review. The limitations of the two methods are different, so that, where the signals conflict, panels may be forced to examine the integrity of their judgement.

A broader role for impact

The main objective, as with most previous reviews, is, however, to reduce the scope for gaming, which is seen as a cause of overhead cost inflation. This committee does not go as far as the review of the 2001 cycle, led by Sir Gareth Roberts, an engineer, which claimed to have eliminated the possibility of gaming… However, they do hope to reduce it significantly by requiring all ‘research-active’ faculty members to be included in the unit return. Critics have already noted that such a requirement will probably reinforce the existing trend towards separating employment and careers in research and teaching. More significantly, they also attack the ‘portability’ of publications.

At present, if someone moves between universities during a REF cycle, their publications are credited to their employer on the REF census date. Stern proposes that they should remain with the employer where the work was carried out. This will eliminate the obvious abuse of buying international scholars on short-term, part-time contracts for the census date. In practice this has not actually been a problem: As Athene Donald points out – and this is also my experience – panels are well able to spot this and adjust their scores to compensate. It has been widely noted that this rule would encounter serious problems in deciding where publications should be attributed, particularly in disciplines where there is a long gap between contracting and publishing work. It may also specifically disadvantage early career researchers who cannot bring their publications to a tenure-able appointment in a different institution. This will limit the kind of mobility that has traditionally been thought important in many STEM disciplines, where junior researchers accumulate experience in a number of laboratories and acquire a repertoire of skills and techniques.

Such a lock-in does, however, have more substantial systemic consequences that have been less fully recognized.

Although the REF is an assessment of departments, these are necessarily made up of individuals. University managements have, then, taken the REF process and applied it to a situation for which it was not designed: instead of focusing on 4* departments, they are focused on 4*individuals. This involves the creation of proxies for the subject panels’ peer review, by the use of consultants – like me – and bibliometrics. They have, in effect, developed signals that allow a true market in talent to emerge.

Unfortunately, they do not like the consequences. Lord Stern’s committee of the great and good sneers at these – individual academics seeking to better themselves are accused of ‘rent-seeking’ and being ‘narrowly-motivated’ for behaving like rational economic actors. The end of portability attaches a ball and chain to ensure they remain the prisoners of central university planning and its associated command and control management. However, this misses the very point of the market. It is a way in which talented and productive researchers can secure appropriate recognition and rewards – and push back against assertive managers who would rather build sports centres than properly resource laboratories or libraries. It is a critical check and balance in the system – and that is why the sorts of people who sit on committees like this want to be rid of it.

The outcome, however, is likely to be even more dysfunctional than the present arrangements, particularly when considering the inherently disruptive nature of genuine innovation. Ending portability discourages mobility and locks researchers into their locations for years at a stretch. It is the enemy of internal migration to develop new combinations and configurations of talent. With its particular impact on early-career researchers, it is likely to encourage endogamy in recruitment so that departments gradually ossify around a narrow agenda that is effectively unchallenged over long periods. When taken alongside the expropriation of traditional rights to determine where to publish and how to share data, it becomes a recipe for box-ticking research by 9-to-5 researchers.

Gaming is not eliminated but shifted to the window immediately after the REF, when universities with inside information from panel members can go headhunting. They will be taking a bigger risk in hoping that someone will be as productive in the next cycle as the last but there is no reason to see this as any less disruptive.

We may not like the salaries paid to leading sports stars on all sorts of justice grounds. In a free society, though, this is not a basis for saying that we should obstruct their movement in search of better opportunities and greater respect. The response to Lord Stern’s review reveals that most policy analysts are central planners at heart and that there are a fair number of our colleagues who would rather cut down the tallest poppies than nurture them. Abandoning portability may be something that they have wished for – but they might be surprised by the consequences of getting it.