What Does Trustworthy and Credible MTurk Research Look Like? Recommendations and Checklist
Did you know that the use of MTurk for research purposes has increased more than 20 times in the past 10 years? Are you interested in using MTurk for your own research and want to make sure you do things right? Have you recently read a study that used an MTurk sample and are wondering whether you can trust its results and conclusions? Are you a journal reviewer who needs to provide constructive feedback to authors on how to improve the quality of their MTurk-based research and would like to know how to go about doing so?
Our Journal of Management article, “MTurk Research: Review and Recommendations,” provides answers to each of these questions. We do so based on two separate literature reviews involving more than 600 sources. Our review includes articles and books in management, accounting, consumer behavior, medical research, organizational psychology, political science, sociology and other fields.
What did we find? First, it’s clear that MTurk’s incredible popularity is based on four main benefits. First, you can use a large and diverse participant pool. For example, MTurk allows researchers access to a larger and more demographically diverse participant pool. For example, compared to student samples, MTurkers are older, have more years of relevant work experience, and report greater computer and internet knowledge. Second, you can access and collect data very quickly. For example, most MTurk assignments are completed within 12 hours or less because, at any given time, there are about 7,300 MTurkers available. Third, MTurk is quite inexpensive compared to other data collection methods. For example, recruiting 100 participants for a survey that takes 10 minutes to complete costs around $120 plus a 20 percent fee for MTurk—a total of only $145. Finally, MTurk gives you flexibility in terms of research design choices. For example, MTurk can be used to implement experimental, passive observation, quasi-experimental, and longitudinal research designs.
But, despite these fabulous benefits, our reviews uncovered 10 critical validity-related challenges that researchers using MTurk face and must address for conclusions to be credible: (1) inattention, (2) self-misrepresentation, (3) self-selection bias, (4) high attrition, (5) inconsistent English language fluency, (6) non-naiveté, (7) growth of MTurker communities, (8) vulnerability to web robots (or “bots”), (9) social desirability bias, and (10) perceived researcher unfairness.
Our article offers evidence-based recommendations for addressing each of the 10 challenges. We took great care in creating this checklist with recommendations that are practical and actionable—not just “pie in the sky.” For example, to address the threats of inattention and vulnerability to web robots, we suggest including a CAPTCHA verification question—a question where MTurkers provide a self-report of response effort, as well as a question requiring an open-ended response. As a second example, to gauge social desirability, we recommend repeating pertinent questions at the end of the study that make explicit the desired response, and contrasting participant answers to the same questions as when presented earlier. As yet another example of solutions, to counter the threat of careless or insufficient effort responding, we recommend analyzing response times and excluding participants who completed the task in less than one or two seconds per item. We offer similarly detailed and actionable solutions for dealing with each of the 10 critical challenges.
Our article is useful for those who want to use MTurk for the first time, as well as for those who have already used MTurk but wish to improve the quality of their future research. Also, journal editors and reviewers can use the checklist to evaluate a submission and provide developmental feedback to increase transparency and reproducibility. In addition, consumers of MTurk research, including other researchers but also practitioners and policy makers, can use our checklist to understand the extent to which a study has followed best practices and therefore can be trusted.
Overall, we hope our Journal of Management article will serve as a catalyst for more robust, reproducible, and trustworthy MTurk-based research in management and other fields. Please do get in touch with us if you have any questions!
