Coronavirus UK – Models or Crystal Balls?
As far back as we have records, humans have tried to predict the future. Why? Because we want to know that we are doing the right thing in the present. Are our actions today going to have the desired results? Can we avert some hazard or danger by making an appropriate decision here and now? Some societies turn to prayer, divination or oracles. Others to tarot cards or crystal balls. In the modern world, much of that function is fulfilled by mathematical models. Is this new technology of forecasting really an upgrade?
In many contexts, it clearly is. Where we are using the mathematics to describe natural systems with minimal human intervention, models can save a great deal of time and effort in making reasonably accurate predictions – about the behaviour of materials under stress, about the effects of environmental chemicals on cells, even about whether it will rain tomorrow. All weather gods are diminished by meteorology. There are two reasons for this. First, the components of the systems do not have agency in the same sense as humans. Second the use of measurement by fiat is uncontested. What does this mean? ‘Fiat’ comes from the Latin for ‘let it be done’. In these contexts, measurements rest on an agreement among scientists about how to assign a number to some state of an object or a process. This agreement is often encoded in a ‘black box’, a measuring tool that every scientist in the field has adopted to create a common currency.
When we come to human systems, the application of the same mathematical techniques is much more problematic. First, human agency means that you cannot just draw a straight line into the future and assume that nothing will change. In the field of transport planning, John Adams conjured up the memorable image of models projecting a future where, by 2205, the whole of Southeast England would be a plain covered in tarmac with lorries roaming endlessly across it, looking for consumables.
Second, it is much harder to establish a common currency for measurement. Again, there are practical solutions to limit the uncertainty. One is to use the law to define what counts as an instance of X, like a birth or a death, and to provide a mechanism, judicial interpretation, to impose a definitive meaning, if required. Short of this, however, all numbers that go into models of social systems rest ultimately on human judgements and actions. All quantitative data start from qualitative observations.
It is important to stress that this does not mean that modelling exercises are invalid or useless. This would perpetuate the sociological nihilism of the 1960s and 1970s, when we first recognized the social construction of all quantitative data. However, it does mean that we need to understand how the numbers that go into models are created, for modellers to display a degree of humility about the inferences that can be drawn, and for policy-makers to use their own judgements about the consequences that might flow from them. As my friends in computer science say: ‘Garbage in, garbage out’.
Let me work through an example relevant to the COVID-19 pandemic. There is a widespread belief that the UK government was spooked in mid-March by projections of the scale of likely deaths resulting from inserting Italian data into a model originally designed around influenza and initially modified by the use of Chinese data. However, the model is very likely to have exaggerated the risks because its authors did not take sufficient account of how the Italian data were produced. Let me explain this – based on public data and interactions with Italian medical sociologists.
The first point to note is that the mortality data that seems to have been used in the model described deaths in hospital. These are relatively easy to count and yield data in real-time. Looking at all deaths introduces a significant lag in most countries because of the delays inherent in the process of registration and recording. However, hospital death figures are the outcome of a complex social organization that can introduce important biases.
Criminologists sometimes say that the most influential person in the criminal justice system is the beat officer who decides whether to turn an interaction with the public into an arrest. Everything else is built on this. In the same way, the significance of hospital deaths starts with the decisions to refer and to admit. Italy has about twice as many hospital beds (and ITU beds) per 10,000 population than the UK does. Typically, these run at 80 per cent capacity, where the NHS runs closer to 100 per cent. This means that admission thresholds to both are likely to be lower than in the UK. If there is capacity, why not fill it up? Robert Evans, a leading health economist, described this as supply-induced demand. Do Italian doctors need to evaluate the decision to admit a patient or use a critical care bed as stringently as UK doctors do – every day of the year? We might also want to ask whether Italian hospitals are staffed to run at 100 per cent, as opposed to having a margin on bed spaces.
Consider also the culture around health care in Italy. Sociologists studying cancer care in that country are frustrated by the scale of aggressive interventions that bring little benefit, and the limited investment in end-of-life care. They tend to see this as a co-production by doctors and relatives. It does not give much weight to the interests of sick people themselves in knowing what their condition is, what its prognosis might be, and whether there might be gentler ways to die. If this is transferable to COVID-19, it is not hard to see how health professionals might be highly stressed by additional workload and triage decisions that they are not accustomed to making. In contrast, NHS health professionals are well used to working under pressure, recognizing the limits of medicine, and weighing the quality of death against the possible extension of life. Is it kinder for an older person to die peacefully in familiar surroundings rather than survive another two or three days under aggressive intensive care?
When we look at hospital deaths and system stress in Italy, then, we are looking at a social organization of dying that tends to steer deaths into hospital and to make heavy resource demands in the course of the dying process. If the numbers are simply dropped into a model, they will produce significant overestimates of the impact on health systems that operate in different ways, with a greater emphasis on primary and palliative care. Hospital deaths will represent a greater share of all COVID-19 deaths in Italy than in countries where deaths in a family home or care home are more customary. The resource demands are likely to be more intensive than in countries where there is greater concern with the quality of death than the marginal extension of life. These factors can only be captured imperfectly within a model but are essential to understanding how to interpret it as a basis for policy. They need to be repeated for each of the lines on any graph comparing international mortality data. The numbers do not speak for themselves.
