Complex modelling, COVID-19 and the case for open systems

Policy makers need to act quickly to better understand complex modelling

When a new and dangerous disease emerges, governments do not have time to wait until clear data about the disease is available and cannot rely that this disease will be similar to others in the past – they have to take immediate action if they are to limit the death toll. The data coming in about COVID19 is complex: composed of different sources that collect the data in different ways, there are many problems the data, and the data that matters – measuring those aspects that will result in deaths (i.e. who is infected) is largely unknown. So what is a government to do? Where can it turn to for informed advice?

The direction that the UK government took, is to turn to modellers – those that produce complex computational models – to make sense of the data in order to give them some idea as what is happening and what is to come. The team of epidemiologists under Neil Ferguson at Imperial College adapted a model that had been developed 13+ years ago to model flu pandemics to the new virus [1]. It was their evaluation of the data coming from Italy using this model that alerted the Government to the size of the epidemic, which would have overwhelmed the health service. This caused the Government to rapidly change their policies, and introduce immediate and drastic social distancing measures. These measures will not cause a significant reduction in terms of how many catch the disease, but slows the infection rate down so its spread happens over a much longer time period. If it slows sufficiently we may have enough intensive care, ventilator beds for those with the most serious symptoms, and hence keep more of those people alive. At the time of writing this, it is not yet clear whether this will succeed.

Being a practitioner myself, I know there are lots of difficulties with such complex models. The Imperial model runs to “thousands of lines of undocumented” computer code [2]. Such models are very hard to check completely, since many of the parts can interact in surprising and unpredictable ways. They also, inevitably, encode assumptions about the disease and how people are spreading it, because the facts are not known. These difficulties are fundamental to this kind of modelling and are not the fault of the Imperial team – they are rapidly reacting to the situation and doing the best they can. Their results are not uncontested – there are many teams racing to model this, all using slightly different assumptions and levels of sophistication in their models – and they do not all agree on the implications.

The interface between such researchers and those involved in policy is not an easy one. In 2012 the UK government ran a competition for who would run a particular service (the InterCity West Coast rail franchise). Civil servants modelled the various bids to assess its costs and benefits and a winner of the competition announced as a result. Unfortunately for the government, this decision was challenged in the courts and it turned out that there were mistakes in the modelling, and the decision was retracted. This caused the UK government to review all its modelling work and, eventually, issued a standard (the “Aqua Book”) [3] for all internal modelling work in order to improve its quality.

It is not even clear that modelling always helps. On the 2nd July 1992 Canada’s fisheries minister, John Crosbie, placed a moratorium on all cod fishing off the northeast coast of Newfoundland and Labrador. That day 30,000 people lost their jobs and hundreds of years fishing for cod off those coasts ended. The cod were declared commercially extinct. The subsequent Harris Commission report [4] into the reasons for this found that the fisheries department’s estimates of stock strength relied upon data, methodologies, and models of such poor quality as to be essentially useless as a ‘rational’ basis for management or commercial planning. One of its conclusions was “…that weaknesses in scientific management and the peer review process permitted this to happen.” (p.2). It seems that the fishery researchers and management had become too close, resulting in assumptions being uncritically shared and many sources of evidence excluded.

As we become more interconnected and our collective life more complicated, we are going to have to increasingly rely on complex modelling of this type. As it says in a recent overview of such modelling published by the Government Office for Science [5] “Just as physical tools and machines extend our physical abilities, models extend our mental abilities, enabling us to understand and control systems beyond our direct intellectual reach”. We are going to have to improve the practice of modelling, but also the interface between modellers and policy actors and, of course, between modellers and the public. Four such suggestions are as follows:

* Modellers will have to get used to making their models openly available to others to check and play with at an early stage – that way it is not just a single team or model that is relied upon but the efforts of a whole community of researchers (in a similar manner to how the software that runs the internet was built).
* It should be standard government policy that they insist that any model that significantly influences important decisions be made open, so that others can see the assumptions that these are base upon and critique them. In other words move towards democratically accountable modelling.
* Policy actors and the public should develop some basic knowledge about such modelling so they know when it can be relied upon and for what kinds of purpose. For example, it is rare that these kinds of model predict the future well, but they can be very useful in understanding what is happening and what might go wrong as a result of interventions [6].
* Ways of making detailed, individual data available to researchers in responsible ways in crisis situations needs to be worked out and routinely implemented. Producing a stream of good quality data takes time, so this needs to be in place *before* a crisis occurs.

Professor Bruce Edmonds is Professor of Social Simulation at the Manchester Metropolitan University Business School and Director of the Centre for Policy Modelling (CfPM) there. He is an expert in agent-based social simulation applied to issues of practical and policy relevance. 

Notes and references:

1. https://www.imperial.ac.uk/mrc-global-infectious-disease-analysis/covid-19/

2. Neil Ferguson tweeted the following on 22 March, 2020, 10:13PM: “I’m conscious that lots of people would like to see and run the pandemic simulation code we are using to model control measures against COVID-19. To explain the background – I wrote the code (thousands of lines of undocumented C) 13+ years ago to model flu pandemics…”.

3. https://gov.uk/government/publications/the-aqua-book-guidance-on-producing-quality-analysis-for-government

4. L. Harris, “Independent Review of the State of the Northern Cod Stock Executive summary and recommendations,” 1990.

5. https://www.gov.uk/government/publications/computational-modelling-blackett-review

6. https://cfpm.org/discussionpapers/236