In this second response to Ziyad Marar’s thought piece “On Measuring Social Science Impact” from Organizational Studies, Anne-Wil Harzing, professor of international management and staff development lead at Middlesex University Business School, sets the stage for further discussion by defining impact in terms of progressing scientific knowledge, developing critical thinking, and addressing societal problems. 

It should be easy, shouldn’t it? Impact is impact is impact. Well, no… When talking about research impact, academics often have very different understandings of the concept. Worse still, they might not even be aware that different interpretations are possible.

As a result, any discussion about research impact soon descends into a Babylonian speech confusion. It also means that many academics are struggling to show the impact of their research, for instance when making a case for promotion (see: Academic promotion tips (1) – What makes a successful application). In this post, I will provide a brief clarification of the concept of research impact and show how it differs by academic role.  

What is impact? 

The Oxford dictionary defines impact as “a marked effect or influence.” Research impact thus means that our research has affected or influenced something or someone. Unfortunately, this immediately throws up even more questions:  

• Whom has it impacted, i.e., who is the target audience?  
• How has it made an impact, i.e., what was its ultimate goal?  
• Through what means has this impact occurred, i.e., what are the primary outlets?  
• How do we know this impact has occurred, i.e., how can we measure it?     

The answer to these four questions very much depends on the specific academic role we are looking at (see table). 

How research impact differs by academic role 

Research role 

In the research role of our academic jobs, our target audience is other academics, and our ultimate goal is to progress scientific knowledge by incorporating our work in the body of scholarly knowledge.  

We connect with other academics through academic journals, research monographs, and conferences. The preferred types of outlets are highly dependent on disciplinary norms and preferences. In the Life Sciences and Natural Sciences academic journals “rule.” In the Humanities and some of the Social Sciences, books remain a popular medium (see: Own your place in the world by writing a book). In some Engineering disciplines, conference papers are a popular way of diffusing knowledge quickly.  

Whether or not we have influenced other academics is typically measured by citations in academic journals, books, and conference papers. Although we know that academics are sometimes careless in their referencing (see: Are referencing errors undermining our scholarship and credibility?) and there are many reasons to cite papers (see On the shoulders of giants? Motives to cite in management research), one would normally expect citations to signify at least some level of impact on the citing academic. Citations can also be field normalized to account for differences in citation practices across disciplines (see: From h-index to hIa: The ins and outs of research metrics). 

Teaching role 

As academics, we all have a tremendous impact on our students. Obviously, some of this impact is unrelated to our research. However, in any good university, research feeds into the classroom and students benefit from research-informed teaching, allowing them to develop critical thinking skills.  

As academics, we facilitate this directly through prescribing our own and other academics’ research as course readings. But in many cases, we will need to “translate” our research to make it more accessible for a student audience. We do so through publishing textbooks or articles in practitioner journals such as Harvard Business Review.  

So how do you know whether your research is used beyond your own classroom? You can find out if your research is cited in textbooks by using Google Books. To discover whether your publications (both academic articles and textbooks or practitioner articles) are listed in teaching syllabi, Open Syllabus Explorer (See Open Syllabus Explorer: evidencing research-based teaching?) is an incredibly useful tool. 

Engagement role 

So far, we have discussed the two key functions of any university: research and teaching. But there is also a third function: external engagement. This captures the impact of our research on industry, government, and the public/society at large, with the ultimate goal being to address key societal problems. It is the kind of impact incorporated in Impact Case Studies in the UK Research Excellence Framework (REF), defined as “an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment, or quality of life, beyond academia”. 

Making our research accessible to an audience outside academia typically means “translating” our research for non-academic use by writing articles for practitioner / professional journals and magazines, as well as publishing policy reports. It may involve the use of (social) media, not just to diffuse our already published work, but to allow continued engagement of non-academic audiences in our research. Integration of non-academic stakeholders from inception of research projects is increasingly common in the social sciences. 

Measuring this type of impact is challenging. Recent efforts like Overton – a database that captures policy impact by tracking citations in policy documents – might help. Academics may also be able to evidence changes in government policy or legislation that are linked to their research. A more general case for impact can be made by referring to frameworks such as the United Nations Human Development Index and Sustainable Development Goals.  

Horses for courses 

Different universities have very different strategies for research impact, especially in the research and engagement roles. In the 2021 REF, the national research evaluation in the UK, the field of Business and Management showed a very different ranking of universities depending on which of the constituent criteria you focused on. London Business School, London School of Economics, and University College London excelled in the perceived quality of their publications, but only had an average performance when looking at their societal impact. In contrast, Middlesex University, the University of Westminster – both modern universities – and SOAS (the School of Oriental and African Studies) topped the list for societal impact. However, they only had an average performance when looking at the perceived quality of their publications.  

In sum: measuring impact 

It is almost impossible to measure the ultimate effect for these three areas of research impact – progressing scientific knowledge, developing critical thinking, and addressing societal problems – accurately and comprehensively by using citations or other quantitative metrics alone. On the other hand, relying only on testimonials and a fully narrative approach to establishing research impact is unlikely to convince audiences, either. So in evidencing impact – for instance when making a case for promotion – academics are advised to rely on a combination of metrics, qualitative evidence such as testimonials, and career narratives.

Resources 

How to Improve your research profile, impact, and reputation 

How to write Effective promotion applications 

How to use Social media to support your career