Analyzing a Canadian Approach to Assessing Impact

Consider this fictionalized story that appears in Approaches to Assessing Impacts in the Humanities and Social Sciences, a report published by the Federation for the Humanities and Social Sciences in Canada:

“A senior Canadian psychology professor has spent 15 years researching learning strategies for people facing various cognitive barriers. One day, she receives an unusual invitation: would she consider contributing to a not-for-profit research and social service network focused on skills training for new immigrants? Intrigued, she accepts, and over the next year works with a team of researchers and practitioners to develop teaching strategies designed to help new immigrants succeed in the Canadian labour market. The new strategies are put to work in the skill-training services offered through the network, which then collects data on various learning and employment outcomes – data our protagonist can use the next time she is asked to describe the impacts of her work. Of course, having worked as part of a large team, accounting for the impacts of her individual contribution will be a challenge.”

There are a number of important lessons we can take away from such a story. For instance, a researcher might work 15 years on a problem before a practical application is discovered. A network that connects researchers, practitioners, and users may be far better equipped to track the impacts of research than any one researcher is able to. But even when such a system produces great impact data, it might still not be possible to attribute those impacts back to an individual.

This story is one of four semi-hypothetical case studies we composed for our report to illustrate the many different forms that impact assessment can take in the humanities and social sciences. The characters in the stories are fictional, but the scenarios are all inspired by real activities taking place in Canada’s research community. While each of these stories helps to illustrate various useful lessons for assessing impacts, the message they send as a whole is perhaps the most important: they show just how diverse the world of research can be. This understanding underlies the approaches we recommend to assessing impacts, and leads us to recommend pluralistic and flexible assessment approaches that are able to account for this diversity and complexity.

Impacts in a Canadian context

The federation set out several years ago to help Canadian scholars navigate the thorny issue of impact assessment, recognising that we are not primarily a research organisation. Our goal was therefore not to materially advance the state of knowledge about how to assess scholarly impacts – we’re happy to leave that to the experts – but rather to bring together existing findings from research, help communicate its relevance to our community, and equip our members with knowledge and tools they can use in the challenging conversations about impact assessment that may lie ahead.

This article by Peter Severinson originally appeared on the LSE Impact of Social Sciences blog as “Approaches to assessing impacts in the humanities and social sciences: recommendations from the Canadian research community” and is reposted under the Creative Commons license (CC BY 3.0).

Our approach was based on a few key observations about how the impacts discussion is developing in Canada. First, while conversations about impacts are gaining prominence across the country, they are both diffused and obscured – taking place at different levels across our vast country and frequently out of public view. Canada’s research granting agencies employ and continue to develop various impact-assessment requirements (which, to date, appear to have only a minor effect on research funding decisions). Meanwhile, universities are increasingly including impact assessment in their strategic plans. Both trends are in part a response to increasing pressure from governments for more performance data on publicly funded programmes and institutions.

Our second key observation was that ongoing conversations about impact assessment in Canada are frequently hindered by uncertainty, anxiety, and suspicion (which, we understand, is a state of affairs not unique to Canada!). In particular, scholars in the humanities and social sciences express concerns that an assessment system based on narrow sets of inflexible metrics would skew the research landscape to the detriment of research disciplines whose impacts resist simple quantification. And, to be sure, some concern is warranted. Not knowing what requirements are coming next, it’s easy to imagine the worst, especially when exposed to heated debates taking place in other countries.

We also heard that university administrators and public servants tasked with exploring impact assessment want to develop responsible assessment practices, ones that don’t cause unintended harm. However, they reported being uncertain about how to accomplish this, even after attempts to consult with researchers.

Contributing to a better conversation

Putting these observations together, the federation saw an opportunity to enrich the Canadian conversation about impacts assessment. Our goal has been to summarize some of the key findings from research on scholarly impacts and present them in a way that addresses the hopes and concerns of the research community while also providing guidance to university administrators and public servants.

Our primary observation – as illustrated through our semi-hypothetical case studies – is that scholarly work is highly diverse and that efforts to assess scholarly impacts must account for this diversity. As a result, we recommend flexible and adaptable approaches to assessing impacts. This includes defining the concept of “impact” broadly, employing a diverse mix of impact indicators (both qualitative and quantitative) and ensuring that researchers themselves play a leading role in selecting the indicators that best suit their research.

We also recommend enhancing our ability to assess impacts by improving the ways we work inside and outside of universities. We recommend that research institutions, including universities and government agencies, provide the necessary resources to support the demanding work of impact assessment. We also recommend that assessment approaches recognise the contributions of non-academic partners, which play so vital a role in the pathways between research and impacts.

The approaches we recommend are intentionally broad. Applying these approaches to specific impact-assessment efforts will require work from researchers, scholarly associations, research funders and universities. Our hope is that our report will help these partners work together productively by providing them with some initial common ground based on a shared understanding of the key factors that affect impact assessment.

It’s an exciting time in Canada as these conversations grow and develop – we expect them to produce great things – and the Federation is thrilled to be making a modest contribution.