Academy of Management Report on Measuring Scholarly Impact

To lead development of valid and reliable measures of scholarship evaluation, the Academy of Management (AOM) recently published an empirical report on Measuring and Achieving Scholarly Impact. This post highlights the major findings, including pluralistic measures of impact, and wider audiences for research.

This article by Usha Haley originally appeared on the LSE Impact of Social Sciences blog as “Beyond Impact Factors: an Academy of Management report on measuring scholarly impact” and is reposted under the Creative Commons license (CC BY 3.0).

With about 20,000 members worldwide, the AOM forms the preeminent scholarly association in management, publishing some of the most-reputed academic journals. The project had two interrelated parts: a qualitative study, and quantitative all-Academy survey on scholarly impact and its meaning to the AOM’s constituencies. This report includes the first survey of membership from a major academic association on what constitutes scholarly impact and which stakeholders have importance for scholarly research.

We began through open-ended, in-depth interviews with 30 members identified by the AOM’s Board of Governors as highly impactful (e.g. journal editors, AOM Presidents, and Fellows). The subsequent membership survey had a response rate of 19%. Despite their ubiquity as measures of scholarly impact, the majority of respondents (60%) indicated that journal rankings and lists, including Impact Factors, probably did not (20%), definitely did not (8%), or might or might not (32%) reflect scholarly impact. A minority (about 41%) indicated that rankings and lists definitely reflected (7%) or probably reflected (34%) scholarly impact.

Conversely, the top five indicators of scholarly impact on a five-point scale (1 = very unimportant, to 5 = very important) were:

• Scholarly articles in top-tier journals – 4.49
• Scholarly citations to research – 4.21
• Scholarly books – 3.94
• Competitive research grants – 3.93
• Articles in practitioner-oriented/industry publications – 3.88.

Indicating the historical focus on internal audiences for academic research, on a five-point scale (1 = very unimportant, to 5 = very important), respondents saw the top five research audiences as:

• Other academics in management – 4.48
• Top management and decision-makers in companies – 4.26
• Government and policymakers – 4.08
• Other academics in the social sciences – 4.06
• Students – 4.0.

Yet, about 54% of the survey’s respondents considered impact on practice as either strongly important (31%) or intensely important (23%); only 7% viewed impact on practice as not at all important as a component of scholarly impact.

Similarly, about 46% of survey respondents considered impact on government policy as either strongly important (27%) or intensely important (19%); only 10% viewed impact on government policy as not at all important as a component of scholarly impact.

Though more difficult to publish, about 59% of the respondents viewed interdisciplinary research, as probably more impactful (31%) or definitely more impactful (28%) than research that draws on one discipline; just 4% of the membership viewed interdisciplinary research as definitely not more impactful than research drawing on one discipline.

Respondents overwhelmingly saw institutional support as very strong for publications in top-tier journals, with other activities receiving far less, if any, support. On a five-point scale (1 = very unimportant, to 5 = very important), respondents ranked the top five beneficiaries of institutional support as:

• Publications in top-tier journals – 4.54
• Scholarly citations to research – 3.76
• Obtaining research grants – 3.64
• Published books – 3.07
• Publications in practitioner journals – 2.84.

Most respondents (47%) saw their institutions as sometimes supporting their personal pursuit of scholarly impact. About 38% said their institution supported their pursuit almost every time (27%) or every time (11%). About 16% indicated their institutions almost never supported (13%) or never supported (3%) their pursuit of scholarly impact.

On the influence of the field, generally, the AOM’s membership thought that management research had been somewhat influential, but the greatest influence had been on other management academics including what they currently research and will research and teach. On a five-point scale (1 = very unimportant, to 5 = very important), the most influence was on:

• Management theorising – 3.91
• Teaching – 3.63
• Future research practice – 3.59
• Management policy and practice in large enterprises – 2.84
• Students’ career decisions – 2.64.

The qualitative data showed persistent themes of high concern from senior scholars regarding the measures that institutions use to gauge scholarly impact, effects on career development, management research’s value, and societal benefits. Most of the scholars stated that the present system of faculty evaluation and business school rankings led to overreliance on more traditional techniques and methodologies, and what journal editors find acceptable. Some scholars identified that these developments had led to “junk science”, journals as “incestuous outlets for career-aspiring management academics”, underreliance on ideas, community and society, and excessive “balkanization” as management scholars became “angels dancing on a pin head”. Some raised concerns about the universal applicability and acceptance abroad of US faculty evaluation standards and research approaches that diminish scholarly impact. One scholar categorised the spread of US research standards globally as amounting to “imperialism” and a form of “colonialism”, with disregard for context.

Management scholars made several actionable recommendations on moving the field forward from its position of high academic legitimacy for wider social influence. Overall, to measure and to achieve scholarly impact, the report reinforced a need to develop composite measures of scholarly impact, to reduce excessive focus on methodologies and techniques, to increase value placed on developing ideas important to external constituencies, and to introduce more applications of theories to practice. As one scholar stated: “[support of this project] shows that the current Board is trying to fight the tradition of the status quo. The Academy has been so successful. Attendance at our annual meetings is the highest among any professional association. So, we have also become a victim of our own success, and there is little incentive to change. We are now criticised for our lack of relevance – and the Board sees that”.

In conclusion, the AOM report and findings fit into the larger conversation on research and researchers’ roles in society by exploring societal values and influence of research. The report has relevance for membership of the Academy, universities’ administrators, journal editors, tenure and promotion committees, accreditation agencies (e.g. AACSB, EQUIS), grant-giving organisations (e.g. NSF), policymakers, national reviews (e.g. PBRF, REF), and society at large.