How to Write, Evaluate, and Use Methodological Literature Reviews

One of the things we’ve learned from our sister website MethodSpace is the genuine hunger among the social and behavioral scientists for how-to information. Today Business and Management INK brings you an essay from the president-elect of the Academy of Management, Herman Aguinis from the George Washington University Business School’s Department of Management; Ravi S. Ramani, an assistant professor of organizational behavior and human resource management at Morgan State University; and and Nawaf Alabduljader, professor of management at Kuwait University-College of Business Administration that, as its title proclaims, offers best practices for methodological literature reviews.

The essay is based on the trio’s paper in the journal Organizational Research Methods, and both the paper and the essay make clear that despite some established conventions and a clear need for clear, thorough and transparent methods literature reviews to avoid questionable research practices, mastery of this technique is widely lacking and concrete advice is necessary. The paper’s abstract and video abstract appears below, followed by the essay. (And yes, we’re reposting this essay on MethodSpace!)

We categorized and content-analyzed 168 methodological literature reviews published in 42 management and applied psychology journals. First, our categorization uncovered that the majority of published reviews (i.e., 85.10%) belong in three categories (i.e., critical, narrative, and descriptive reviews), which points to opportunities and promising directions for additional types of methodological literature reviews in the future (e.g., meta-analytic and umbrella reviews). Second, our content analysis uncovered implicit features of published methodological literature reviews. Based on the results of our content analysis, we created a checklist of actionable recommendations regarding 10 components to include to enhance a methodological literature review’s thoroughness, clarity, and ultimately, usefulness. Third, we describe choices and judgment calls in published reviews and provide detailed explications of exemplars that illustrate how those choices and judgment calls can be made explicit. Overall, our article offers recommendations that are useful for three methodological literature review stakeholder groups: producers (i.e., potential authors), evaluators (i.e., journal editors and reviewers), and users (i.e., substantive researchers interested in learning about a particular methodological issue and individuals tasked with training the next generation of scholars).

Groundbreaking discoveries in business management and many other fields are fueled by methodological innovations. From increases in computing power and the emergence of Big Data to the introduction of sophisticated new analytic techniques, we now have a smorgasbord of new methods and techniques to collect and analyze data. But, the picture is not all rosy. There is an overwhelming and even frantic pace of methodological developments, decreased funding for doctoral programs and researcher training in general, and increased competition to publish in high-prestige journals from around the world. As a result, many researchers find themselves struggling to stay up-to-date on methods-related issues.

So how do we researchers keep up with these advances? We often rely on methodological literature reviews. These are reviews that summarize a methodological issue and provide best-practice recommendations. Simply put, they describe “how to do things right.” They are valuable because researchers use them to understand how to apply new and existing methods, instructors rely on them to train doctoral students, and they are a resource for journal editors and reviewers to identify potentially questionable research practices (QRPs) in the manuscripts they evaluate. But, despite their popularity, the details of how to write and also evaluate methodological literature reviews are not clear. Our project started when we realized there was a need for guidance on how to write and evaluate methodological literature reviews.

Our article published in Organizational Research Methods titled “Best-Practice Recommendations for Producers, Evaluators, and Users of Methodological Literature Reviews” addresses this need. We spell out what goes into preparing a methodological literature review and what criteria can be used to judge them. We describe different types of reviews and their purposes, and uncover hidden features in published methodological literature reviews. Also, on a very practical note, our article includes a table with a checklist of actionable recommendations on how to write and evaluate a methodological literature review.

Our analysis of 168 methodological literature reviews published in 42 different journals revealed some interesting and surprising facts. For example, there are many different ways to write a successful review—one that is published in a highly visible and prestigious journal. And, despite increased attention to issues of research transparency and reproducibility, clear reporting about the process used to conduct a review is mostly absent. We also recommend specific tools (e.g., tables, specific type of language) that can be used to increase readability and accessibility, but which are underutilized.

Overall, our article helps improve the thoroughness, clarity, and usefulness of methodological literature reviews. Authors and journal editors and reviewers can use our recommendations to enhance transparency and clarity, and avoid potential QRPs. Readers of methodological literature reviews can use our recommendations to evaluate the trustworthiness of a review, and use our suggestions to perhaps even author one themselves!

We look forward to hearing your reactions to our article and hope that it will serve as a catalyst to further enhance the quality of literature reviews of methods in management, applied psychology, and other fields.