Here Are the Blocks You Need to Tell Your Impact Story
The increasing pressure placed upon research organisations of all kinds (not just universities) to demonstrate impact is often passed along to researchers and research managers without the support of an accompanying cultural change within the institution.
With this in mind, Forest Research, the government research agency that supports the United Kingdom’s forestry sector, has committed to promote a culture of knowledge exchange and interdisciplinarity in its research. This mission stemmed from a 2012 external impact evaluation which recommended directing attention toward helping researchers generate and record impacts. In addition to program activities such as training workshops and small-group conversations, the initiative identified an additional challenge: that of demystifying and operationalizing impact evaluation in a coherent and accessible way that nonetheless allows for diversity, subtlety and uncertainty.
A new evaluation framework: building blocks
To undertake this almost contradictory challenge, we developed a new evaluation framework that we tested across multiple projects in a pilot exercise. Working with the draft framework made it possible to produce a dozen full impact case studies from initial observations. Analysis of this exercise indicated that the components of the framework were sufficiently clear and comprehensive for researchers and stakeholders to understand and use in constructing a range of very different narratives.
We envisioned the framework as a set of building blocks which could be combined in multiple ways to tell almost any impact story, each shedding light on the complex interactions between research, policy and practice. In fact, when we gave a demo at the 2019 UK Knowledge Mobilisation Forum, we represented each component with different wooden tiles (fitting for work with a forest research agency!) and then used them to tell two contrasting impact case study narratives, showing how we could manipulate the tiles into different configurations to reveal the relationships between them. The audience seemed to appreciate our main messages of dynamics – in terms of multi-directional interactions unfolding in varying sequences – and heterogeneity – in terms of impact types, causes and stakeholders – as an alternative or complement to conventional linear logic models that tend to dominate the field of impact evaluation.
What are the building blocks?
There are: five types of impacts; five broad categories of stakeholders; and eight causal factors, along with a set of over-arching questions.
Five impact types
From previous impact evaluations and other research, we had found that providing a range of impact types to consider often makes people feel ‘liberated’ and ‘authorized’ to go beyond conventionally-sought instrumental changes when searching for their impacts. The five impact types are:
1) Instrumental: changes to plans, decisions, behaviors, practices, actions, policies
2) Conceptual: changes to knowledge, awareness, attitudes, emotions
3) Capacity-building: changes to skills and expertise
4) Enduring connectivity: changes to the number and quality of relationships and trust
5) Culture/attitudes towards knowledge exchange, and research impact itself
There is no fixed hierarchy or linear sequence inherent in this typology, and all have value in their own right, although the last two, process-based impacts may make other sorts of impacts more likely.
Asking ‘How do we know?’ leads towards gathering of evidence, including recognition of telling indicators of change.
Five categories of stakeholders
To stimulate further definition (or mapping) for a particular situation, the framework offers five broad categories of stakeholders, which of course can be subdivided further, and may influence as well as be influenced by the research process and its outputs.
- Policy-makers: government agencies and regulatory bodies, local, national and international
2) Practitioners: public, private, NGOs
3) Communities: of places or interest, general public
4) Researchers: within and beyond the project and the institution
5) Other
Eight causal factors
Because impact generation is a dynamic process, it is important to ask: ‘Why/how did changes occur?’ The framework offers eight causal factors:
1) Problem-framing: Level of importance; tractability of the problem; active negotiation of research questions; appropriateness of research design.
2) Research management: Research culture; integration between disciplines and teams; promotion of research services; planning; strategy.
3) Inputs: Funding; staff capacity and turnover; legacy of previous work; access to equipment and resources.
4) Outputs: Quality and usefulness of content; appropriate format.
5) Dissemination: Targeted and efficient delivery of outputs to users and other audiences.
6) Engagement: Level and quality of interaction with users and other stakeholders; co-production of knowledge; collaboration during design, dissemination and uptake of outputs.
7) Users: Influence of knowledge intermediaries, e.g. ‘champions’ and user groups; incentives and reinforcement to encourage uptake.
8) Context: Societal, political, economic, biophysical, climate and geographical factors.
Reflective questions
The questions underlying the framework are: ‘Who or what changed? (And how do we know?); Why or how did changes occur? and What lessons can be learned?’
These are centered around two questions which encourage the framework’s users to step back and think about what lessons have been learned regarding impact identification and generation, to plan actions and improve future efforts:
1) What worked? What could (or should) have been done differently?
2) What could (or should) be done in the future?
The framework can be used to stimulate critical reflection by individual researchers or teams, ideally involving end-users and other stakeholders, throughout the project cycle, at the planning, mid-course and final stages.
From simplicity to complexity
When assessing impacts, there can be a temptation to package notions of impact into readily countable and comparable units which come about as a result of a simple linear sequence of events. Instead, we tried to encourage researchers to see impact as complex, developing over often lengthy periods of time, in a dynamic context in which different causal factors, stakeholders and impacts themselves interact. Ultimately, impacts are highly heterogenous and travel different developmental pathways. Their stories are multi-dimensional and based on complex interactions.
Forestry was an ideal field in which to develop a framework of this kind, as it draws together a ‘real world’ orientation with an interdisciplinary approach spanning the natural and social sciences. However, we see the framework as having a wider application and being flexible enough to meet the needs of other disciplines and research organisations.
The ‘building blocks’ (impact types, stakeholders and causal factors) we identified can be assembled in multiple combinations to construct nuanced impact narratives in a way that is helpful for accountability (e.g. for funders), communication (e.g. for stakeholders) and learning (e.g. for researchers/managers).
Whatever their organisation, sector or country, we encourage readers to try it out and judge for themselves!
