Business and Management INK

A Quick Examination of Existing Academic Impact Metrics and Concerns in Business Education

January 30, 2023 3608

A new white paper from SAGE Business examines existing bibliometrics and institutional reward structures at play within business schools. We aim to move the dial toward ways in which societal impact could become central to the assessment of business and management research.

Usha Haley, left, and Andrew Jack

In a pioneering survey of its membership in 2017, the Academy of Management asked its worldwide membership what constituted scholarly impact. (With about 20,000 members worldwide, the AOM is the preeminent scholarly association in management. The subsequent membership survey had a response rate of 19 percent, or 700 respondents.)

Despite the ubiquity of journal impact factors as measures of scholarly impact, the majority of respondents (60 percent) indicated that journal rankings and lists, including impact factors, probably or definitely did not or might or might not reflect scholarly impact. Conversely, the top five indicators of scholarly impact were: Scholarly articles in top-tier journals, scholarly citations to research, scholarly books, competitive research grants, and articles in practitioner-oriented/industry publications.

Indicating the historical focus on internal audiences for academic research, respondents to the AOM survey saw the top five research audiences as other academics in management, top management and decision-makers in companies, government and policymakers, other academics in the social sciences, and students.

This is the first of seven posts excerpting and adapting the SAGE Business white paper “Measuring Societal Impact in Business & Management Research: From Challenges to Change.” The white paper includes a core section, written by Usha Haley and Andrew Jack, and complementary essays penned by Ben McLeish and Mike Taylor of Altmetric / Digital Science, Sir Cary L. Cooper at the Alliance Manchester Business School, Renate E. Meyer, WU Vienna / Copenhagen Business School Maura L. Scott, Florida State University.

On the influence of the field, generally, respondents thought that management research had been somewhat influential, but the greatest influence had been on other management academics, including what they currently research and will research and teach. Yet, about 54 percent of the survey’s respondents considered impact on practice as either strongly or intensely important. Similarly, about 46 percent considered impact on government policy as either strongly or intensely important. Though more difficult to publish, about 59 percent viewed interdisciplinary research as probably more or definitely more impactful than research that draws on one discipline. Respondents overwhelmingly saw institutional support as very strong for publications in top-tier journals, with other activities receiving far less, if any, support. Still, only 38 percent said their own institutions supported research with external impact.

Similarly, a global survey undertaken by SAGE in 2021 of social science academics (the first author was involved in drafting this survey and analyzing results), found that of the 373 global respondents in business and management (most from the United States, United Kingdom and India), 81 percent said that it was either important or highly important that their research have value outside academics, and 62 percent stated that their peers felt the same way about their own research. 

Two bar charts, the first showing the high Importance to me of my research having an effect outside academics (n=373 business academics, and the second, that academics strongly agree that Ultimately the goal of my research is to make a positive impact on society
Two bar charts, the first showing How my institution rewards efforts to apply research outside academia n=373 business academics), and the second the high Importance of research getting published in highly cited journals/journals with high impact factors (as
measured by Journal Citation Reports)

Ninety-four percent of the respondents either agreed or strongly agreed that ultimately the goal of their research is to make a positive impact on society. Yet, when asked if their institutions rewarded these efforts to apply research outside academia, only about a third thought they did so. Only 36 percent said that having external impact mattered for tenure, 34 percent said for awards, 32 percent said for funding further research, and, 30 percent for other research resources. Thirty-one percent of the respondents said their institutions provided no rewards or acknowledgments for having external impact through their research. Unsurprisingly, 63 percent either agreed or strongly agreed that ultimately the goal of their research is career advancement. Seventy-seven percent of the respondents stated that publishing their research in a highly cited journal with a high impact factor was either important or very important.

A 2022 poll of 9,609 academics by Times Higher Education found similar results. Aside from personal interactions with the researchers, the perceived quality of the journals where the researchers published their findings constituted the most influential factor when forming opinions of academic standing, according to 49 percent of the respondents. Citation metrics had lower support. Only 24 percent of respondents said a scholar’s h-index and other similar measures were important, and only 5 percent said they constituted the most crucial factor. These findings regarding citations could arise because of increasing awareness of gaming the system through self-citations, citations that speak negatively of the articles, forced citations by journal editors and reviewers, and citations made by researchers who haven’t even read the articles (see Haley, 2021). Additionally, citation counts vary by source—for instance, those in Google Scholar differ from those in Clarivate.

Few measures have exerted greater influence than the journal impact factor, or JIF. Initially developed to help librarians to purchase journals, Clarivate’s JIF has transmogrified into an evaluation of the quality of individual publications and of individual researchers.

The measure has come under extensive scrutiny and criticism: as an inaccurate estimate of citations of any article within a specific journal, as easy to manipulate, and with no associations to objective measures of quality. A decade ago, the San Francisco Declaration on Research Assessment (DORA), critiqued using impact factors as a surrogate measure of quality for individual articles and researchers. DORA currently has 20,000 individual and 2,600 institutional signatories worldwide.

The AOM survey and FT awards (see Jack, 2022) have identified interdisciplinary research as more impactful than single discipline research as it incorporates diverse perspectives and experiences. Yet, interdisciplinary research is also more difficult to publish. Ricardo Fini, Julien Jourdan, Markus Perkmann, and Laura Toschi argued last year in Organization Science that interdisciplinary researchers and research may threaten disciplines’ and evaluators’ distinctiveness and knowledge domains, and hence, evaluators may penalize them. High-performing, interdisciplinary researchers appeared to suffer the greatest penalties in small and distinctive academic discipline where evaluators appeared as representative members of their disciplines. Fini et al. argued that attempts to maintain social boundaries contributed to the relative lack of interdisciplinary research.


Upcoming Articles in This Series

“A Quick Examination of Existing Academic Impact Metrics and Concerns in Business Education” | Usha Haley and Andrew Jack

“How Might Societal Impact be Recognized within an FT Top 50 Journal?” | Renate E. Meyer

“A Decades-Long Journey of Marketing and Public Policy Research to Support the Greater Good” | Maura L. Scott

“Why Don’t Business Schools Publish More Impactful Research?” | Ben McLeish and Mike Taylor

“Some Opportunities for Future Business & Management Research: Employee Health and Well-Being” | Sir Cary L. Cooper

“Efforts to Turn the Tide” | Usha Haley and Andrew Jack

“Medium And Short-term Recommendations to Move Forward” | Usha Haley and Andrew Jack

Usha Haley is W. Frank Barton Distinguished Chair in International Business, and professor of Management at Wichita State University. She is also director of the Center for International Business Advancement and chair of the World Trade Council of Wichita. She has over 400 publications and presentations on non-market economies, subsidies, multinational corporations, emerging markets, trade, strategy, and scholarly impact, including eight books. Her latest book is Impact and the Management Researcher. Andrew Jack is a global education editor for the Financial Times, writing on educational issues around the world and editorial lead for the free FT schools program. He was previously head of curated content, deputy editor of the big read section, pharmaceuticals correspondent, and a foreign correspondent in France and Russia.

View all posts by Usha Haley and Andrew Jack

Related Articles

Boards and Internationalization Speed
Business and Management INK
November 18, 2024

Boards and Internationalization Speed

Read Now
How Managers Can Enhance Trust
Business and Management INK
November 11, 2024

How Managers Can Enhance Trust

Read Now
Tom Burns, 1959-2024: A Pioneer in Learning Development 
Impact
November 5, 2024

Tom Burns, 1959-2024: A Pioneer in Learning Development 

Read Now
The Role of Place in Sustainability
Business and Management INK
October 28, 2024

The Role of Place in Sustainability

Read Now
Turning to Glitter in Management Studies – Why We Should Take ‘Unserious’ Glitter Serious to Understand New Management Practices

Turning to Glitter in Management Studies – Why We Should Take ‘Unserious’ Glitter Serious to Understand New Management Practices

In this article, author Jette Sandager reflects on the inspiration behind her research article, “The sensuous governmentality of glitter: Educating managing women scientists […]

Read Now
Research Assessment, Scientometrics, and Qualitative v. Quantitative Measures

Research Assessment, Scientometrics, and Qualitative v. Quantitative Measures

The creation of the Coalition for Advancing Research Assessment (CoARA) has led to a heated debate on the balance between peer review and evaluative metrics in research assessment regimes. Luciana Balboa, Elizabeth Gadd, Eva Mendez, Janne Pölönen, Karen Stroobants, Erzsebet Toth Cithra and the CoARA Steering Board address these arguments and state CoARA’s commitment to finding ways in which peer review and bibliometrics can be used together responsibly.

Read Now
Paper to Advance Debate on Dual-Process Theories Genuinely Advanced Debate

Paper to Advance Debate on Dual-Process Theories Genuinely Advanced Debate

Sage 984 Impact

Psychologists Jonathan St. B. T. Evans and Keith E. Stanovich have a history of publishing important research papers that resonate for years.

Read Now
0 0 votes
Article Rating
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Newest
Oldest Most Voted
Inline Feedbacks
View all comments