Technology Can Collect and Analyze Evidence for Policy

There is more and more research being produced around the world every day. In total, about 3 million articles are published every year.

That’s a lot of information, and a lot of evidence. But humans are finding it increasingly harder to read, analyse and assess so much data when trying to understand a particular topic, a process called evidence synthesis. This involves converting large bodies of scientific research – articles, reports and data – into reliable and digestible evidence that can inform management or policy.

So if humans might struggle to cope with the increasing volume of evidence needed to build effective, solid policy, what’s the solution? We think technology is the key. With accessible software tools and workflows, machines can be left to do the laborious work so that people can focus on planning, thinking and doing. That’s what prompted two of us (Neal and Martin) to create the Evidence Synthesis Hackathon series.

This initiative was launched in 2017 to bring together world-leading and emerging researchers, practitioners and software developers to produce new Open Source tools and frameworks that support evidence synthesis. There have been three hackathons since then – two in Stockholm, and one in Canberra. They’ve drawn participants from 13 countries on six continents and led to 19 projects being initiated.

One such project is metafor automated reports, which automatically writes methods and results text for a particular statistical model (meta-analysis). This ensures that all the relevant information is included in any report in a consistent and reliable way.

Other projects include tools that help researchers visualise databases of studies to help identify gaps in global knowledge, and those that extract information from documents, like important data that describe the study location or its findings. We have also produced discussion papers that introduce new ways to think about evidence synthesis.

The hackathons, and other digital projects of this nature, are one way of creating a community of practice which together produces freely accessible tools and workflows. This helps to ensure the tools can speak to each other and reduces the risk of lots of different tools being produced that do the same thing.

In practice

Many of the outputs produced at the hackathons are already being used by researchers. One example is EviAtlas. This is a tool for producing maps of evidence unearthed during systematic literature reviews. It converts a database into a set of attractive, interactive figures and tables that show patterns in the “evidence base” and where knowledge gaps and clusters might exist.

It also allows users to produce free, interactive websites displaying the nature of the evidence on a geographical map. This is something that would previously have been expensive and highly complex.

So, for instance if you wanted to know what research had been conducted on the impacts of buffer strips around farmland in temperate ecosystems in Africa, you could quickly and easily explore this interactive map to find out.

Evidence synthesis, much like the primary research studies that the process is based on, can sometimes ignore specialist evidence from Africa – perhaps because researchers do not know the literature landscape as well as that from North America and Europe. Similarly, developing policies based on examples and evidence from developed world contexts doesn’t work for countries with very different contexts. That’s why it’s so encouraging that there’s a growing African presence at the hackathons.

African participation

Earlier in 2019, the Evidence Synthesis Hackathon was able to host three participants from the African continent, thanks to funding from the University of Johannesburg through the Africa Centre for Evidence.

The three participants were integral to the discussions and coding work at the hackathon, and were instrumental in its success.

Two of the participants were experienced software programmers Christopher Penkin and Mandlenkosi Ngwenya. They produced a tool to keep track of and save web-based searches for research. Until now, this has been almost impossible to do in evidence syntheses.

The tool, which is in the final stages of development, is a Chrome browser extension that logs user search information and downloads it into a central database. Internet searches are notoriously difficult to keep track of and report transparently, but this tool does the reporting for you, and also automatically saves all your search results in one place.

This represents a huge step forward in transparency, efficiency and repeatability, and an important gap in the methodology until now. All of this is crucial for rigorous evidence synthesis.

The third attendee from Africa, Witness Mapanga, is an evidence synthesis specialist. He worked with other top researchers on projects to build a brighter future for evidence use in policy. This work is due to be published by the end of the year.

Building communities

We’re planning a number of hackathons in 2019 and beyond – one of them in South Africa in 2020. Doing this will hopefully highlight what resource-constrained environments need from evidence synthesis and what they can produce.

In the long run, hackathons and similar events can be used to build communities of practice: networks of researchers, data scientists, and software developers focussed on driving progress towards a sustainable future. The Evidence Synthesis Hackathon represents a novel but increasingly important part of this new movement.

Mandlenkosi Ngwenya and Christopher Penkin contributed to this article.