Who’s Afraid of the Big Bad Metric?
My, what numerous citations you have!
There is nothing especially wrong with using citation metrics to gauge the state of academic research and the relative position of its players. However, the players need to know the rules of the game, if they are to respond intelligently. To be sure, academics already know that citations operate as market signals by which the fate of their pieces are at least partly determined. Post-peer review citation practices routinely testify to this. So it would be a deep mistake to think that academics are allergic to metrics per se. However, in the past, these metrics were self-organizing and locally enforced.
I am not the first to observe that over the 20th century, academia has devolved into fiefdoms and warlords, whose existence tracks the proliferation of journals. Thus, bibliometricians have revealed that the spontaneous arrangement of citations in the field of published knowledge is constructed as mildly overlapping archipelagos (‘networks’ is a bit euphemistic in this context). But academics would need to approach the task of citation differently in light of the current HEFCE proposal – and this should be welcomed.
This article by Steve Fuller originally appeared on the LSE Impact of Social Sciences blog as “Is the fear of metrics symptomatic of a deeper malaise? Fiefdoms and scapegoats of the academic community” and is reposted under the Creative Commons license (CC BY 3.0).
If academics are indeed preoccupied with turf-marking in the sense so easily revealed by bibliometrics, then it is easy to see how they might be threatened by any global, unifying perspective as suggested by HEFCE, one that would force you to think about who might take your work seriously outside its default sphere of control. Nevertheless, truly interdisciplinary work tends to become more prominent in this global perspective, if you imagine it as gathering a few citations from many fields over a long period rather than many citations from one field over a short period. The matter isn’t usually put this way because ‘peer review’, the procedure by which research is normally judged, is not seen as ‘quantitative’. But in fact it is: It’s just that the number of people who matter and how they’re chosen is more restricted. To take peer review as the norm is ipso facto to mark research as the inherent property of particular fields, rather than as something of potential relevance to the entire academic community, depending on how it responds.
Notice I am not making a larger claim about opening up judgement on academic knowledge claims to the general public. Rather, I am granting the obvious fact that any potentially interesting knowledge claims have cross-disciplinary relevance, in which case the full academic community – not simply specialists in a given field – should be allowed a free say. (Indeed, even now many works in a given field are prominent mainly because of what they say about things outside their nominal field.) To be sure, citations are not perfect indicators of this phenomenon. However, understood as a globally enforced metric, and with the innovations proposed by the ‘altmetrics’ crowd, they are a step in the right direction. And yes, academic citation should be seen as a form of marketing, in which authors attempt to couple their fate with others who not only support their knowledge claims but also enjoy an authority beyond that fact.
More specifically, academics should be trained to think about their citations as investments under conditions of scarcity – that is, exactly like a capital resource. A step in this direction would be for journals to force authors to fractionate their citations, that is, to assume that each author casts only one vote in the overall pool of citations. At the moment, any author effectively casts any number of votes through citation, which implies inter alia that heavy citers have a disproportionate impact on the overall citation count. This results in an unhealthy epistemic dependency culture replete in ‘dummy cites’ to people ‘one is supposed to cite’, regardless of true relevance. (I originally raised this point in 1997, in my Open University book, Science, chapter 4, as something that a Martian might recommend to correct the injustices in Earth’s knowledge system.)
To be sure, in the end, we will need to admit upfront that the move to citations is a move to integrate a proper conception of markets into the internal dynamics of academic knowledge production. Given proper regulation (which should be an outcome of the HEFCE consultation), marketization is doable in a way that is accountable not only to the academics who buy and sell knowledge claims through the medium of publication but also to those – that is, the state, the general public and (where appropriate) private funders – who routinely provide relatively stable conditions for such knowledge claims to be transacted.
In this regard, I despair at the ‘critique’ launched by the international relations scholars, Meera Sabaratnam and Paul Kirby, which has received the endorsement of the Campaign for Public Universities and the British Sociological Association. The telling moment for me of ‘Why Metrics Can’t Measure Research Quality’ appears in point 7, which takes on the high citation count for Samuel Huntington, the Harvard political scientist (and former Jimmy Carter adviser) who predicted a ‘clash of civilizations’ between Islam and the West in the early 1990s.
The problem with such scholars complaining about the high citation rates accorded to Huntington is that they end up sawing off the limb of the tree on which they sit. Much of their own scholarship is precisely predicated on Huntington being so badly wrong, which the authors themselves admit is a major source of Huntington’s high citation rate. However, if we didn’t take citation counts seriously, then it would not be clear why we would wish to pay attention to Huntington’s critics, who usually end their diatribes against Huntington, say, by identifying one ‘green shoot’ of an alternative interpretation that empirically suits a specific locale or simply gesturing for a more open-minded view of the role of culture/religion in political life – to be sure, just as long as it doesn’t veer into extreme relativism or (horribile dictu) libertarianism.
The critics seem to be incapable of either living with Huntington or without him. But this is less a problem of research metrics than of the old academic left’s inability to define a coherent ideal of progressive scholarship, beyond the identification of scapegoats. In that case, the scapegoats deserve all the citations they receive – and the elimination of citation counts is not a solution the academic left’s problems. However, a new positive vision of the future that attracts citations would do the trick.
***
This is a slightly expanded version of my submission to the HEFCE consultation on the use of metrics as a basis for research evaluation.
