What Can REF Teach Us About Impact?

Ever since reporting on the Research Excellence Framework began, a number of debates and controversies have followed. The REF has seemingly presented a number of problems for how academics see their role and how their work is perceived, and valued, by their managers. The new rules of this research assessment instrument appeared allusive for many academics, whilst a number of concerns were raised in relation to how ‘impact’ would be measured and how the REF might shape behaviors within the wider higher education landscape.

This article by Tony Murphy and Daniel Sage originally appeared on the LSE Impact of Social Sciences blog as “Perceptions and ‘impacts’ of the REF: Key aim for next round should be to explore apprehension and minimise anxieties” and is reposted under the Creative Commons license (CC BY 3.0).

As curious academics, we wanted to understand what was being written about the REF and what this might signal in terms of the broader ‘impacts’ of the REF: from individual academics all the way up to the nature of academia itself. In order to do this, we undertook an analysis of the media: collecting media outputs on the REF over a two-year period to identify who was writing about the REF, what issues were being discussed alongside it, and the extent to which reporting was positive or negative.

Our findings raised some interesting ideas about the way in which the REF has been experienced by different types of authors. The results are published in a paper for the Journal of Higher Education Policy and Management. Our first key finding was – perhaps unsurprisingly – that reporting on the REF tended to be negative, regardless of who wrote the article. However, articles by senior academics (most often professors) were the most negative. We also found differences in the level of negativity according to subject area; for example, authors from the natural sciences were significantly more negative than social science authors.

We also found that those based at research intensive institutions were more likely to be writing about the REF in the first place; thus suggesting that they are more likely to have their voices heard. As we neared the REF submissions deadline we found a greater portion of articles citing the REF.  However, as we neared the REF submissions date we also found that reporting on the REF became less negative on an aggregate level – still negative, but not quite as negative as earlier in the cycle. Had academics become more at ease with the process or, perhaps, were they more focused on the task at hand?

Amongst the prominent themes discussed in the articles, ‘impact’ was the most frequent one. Others included ‘funding’ and ‘marketization.’ We also found that ‘institutional inequalities,’ ‘staff and management tensions,’ ‘intellectual freedom,’ ‘dishonest practice,’ ‘staff morale,’ ‘recruitment’ and ‘pressures to publish’ also figured strongly. For each of these themes we found, on an aggregate level, that the discussions around them were negative. The discussions around ‘staff morale’ and ‘staff and management tensions’ were the most negative on our scale. The level of negativity varied according to author background, as well as the issues being discussed.

From this, we argue that the REF has seemingly been experienced by different authors in a number of varying ways; different issues seem to be of concern for different types of authors, and some issues seem to attract a more negative discussion than others. We were particularly interested in the high level of negativity apportioned to professors, and the data seems to link this to a concern for ‘intellectual freedom.’ A further interesting finding is the comparatively heightened concern for ‘impact’ among junior academics and those at ‘teaching intensive’ institutions.

In all, we suggest that such findings offer a sense of some of the issues associated with the REF; they signal the range of concerns academics have around the REF and its processes. The perception that the REF can adversely shape behaviors is particularly noteworthy; we found discussions around ‘intellectual freedom,’ ‘dishonest practice’ and ‘disregard for certain research.’ We argue that this makes it ever-more important that we get it right when thinking about ‘impact’ in research and the pressures academics face, and how this might manifest itself in dubious or even dishonest research practices.

Accountability and transparency are, most academics would agree, vital within the management and funding of universities. Naturally, this extends to research activities. However, future research assessment exercises must be as transparent as possible, with clearly defined and accessible rules. We must also consider the possibility of adverse or even perverse outcomes when defining those rules. Exactly how the next research assessment process will pan-out is hard to predict with any certainty at this point in time.  However, as our research suggests, a key aim of the process should be to explore the broader apprehension that academics approach it with.  And, where possible, strive to minimize such anxieties.

***

For further details please see the original paper: Murphy, T., Sage, D. (forthcoming,2014). “Perceptions of the UK’s Research Excellence Framework 2014: a media analysis.” Journal of Higher Education Policy and Management. 36(6).