Business and Management INK

Answers to 20 Questions About Interrater Reliability and Interrater Agreement

September 26, 2011 1811

James M. LeBreton, Purdue University, and Jenell L. Senter, Wayne State University, published “Answers to 20 Questions About Interrater Reliability and Interrater Agreement” in the October 2008 issue of Organizational Research Methods (ORM). It was the most-frequently read article of July 2011 for ORM, based on calculations from Highwire-hosted articles. Most-read rankings are recalculated at the beginning of the month and are based on full-text and pdf views. Professor LeBreton kindly provided the following responses to the article.

Who is the target audience for this article?

The primary audience is graduate students, faculty, and practitioners working in organizational psychology, human resource management, and organizational behavior. That said, I have received a number of e-mails asking about the paper from colleagues scattered across a wide array of social sciences.

Were there findings that were surprising to you?

Our particular paper was designed to test a priori hypotheses. Instead, we sought to synthesize and integrate roughly 30 years of thinking on issues related to interrater reliability and interrater agreement. So we did not have “findings”, per se. Instead we structured our paper as 20 key questions related to the use of interrater reliability and agreement statistics. We then did our best to answer these questions in a way that would provide others with a clear set of guidelines for using these statistics in their research and work.

How do you see this study influencing future practice?

We hope that our paper provides helpful guidelines for individuals using interrater reliability and agreement statistics in their practice. Practitioners often invoke these statistics when conducting organizational climate or culture studies, performance evaluation studies, or even when examining the quality of rating obtained via a panel of interviewers. Our paper was written to provide important information for practitioners and researchers to help guide a) the selection of the correct agreement or reliability statistic, 2) the correct estimation of the statistic, 3) the correct interpretation of the statistic, and 4) understand how various features of their situation might influence estimates of agreement/reliability (e.g., missing data; number of items on a scale, number of raters/judges).

How does this study fit into your body of work/line of research?

I have been publishing articles that use or refine estimates of interrater agreement and reliability for roughly 10 years. This paper represents an opportunity to reflect on my thinking over these years and integrate it with the thinking of my co-author (Jenell Wittmer-Senter) to arrive at a product that we believe will be helpful to both researchers and practitioners.

How did your paper change during the review process?

The most substantive changes involved providing a more balanced treatment of the rwg coefficient. This is the coefficient that I use in my work and thus we were probably a bit too laudatory in our evaluation. The revised paper presents both the pros and cons of rwg. I still think it is a great way to estimate agreement, it is not without its limitations. Those are now addressed more explicitly in our final paper. We also expanded the set of agreement coefficients we discussed to include awg, AD, and SD.

What, if anything, would you do differently if you could go back and do this study again?

As I noted above, this paper wasn’t structured as a traditional “research study.” Thus, there are particular design or analysis issues I would like to do differently. Overall I am quite pleased with the paper. I believe it has the potential to serve as a helpful resource to individuals wanting to estimate interrater agreement and/or interrater reliability. It was structured as a Q & A paper. We certainly didn’t address all possible questions related to agreement and reliability, but I hope we addressed some of the more pressing ones for individuals who are new to using these statistics.

To learn more about Organizational Research Methods, please click here.

Bookmark and Share

[polldaddy rating=”4667602″]

Business and Management INK puts the spotlight on research published in our more than 100 management and business journals. We feature an inside view of the research that’s being published in top-tier SAGE journals by the authors themselves.

View all posts by Business & Management INK

Related Articles

Boards and Internationalization Speed
Business and Management INK
November 18, 2024

Boards and Internationalization Speed

Read Now
How Managers Can Enhance Trust
Business and Management INK
November 11, 2024

How Managers Can Enhance Trust

Read Now
The Role of Place in Sustainability
Business and Management INK
October 28, 2024

The Role of Place in Sustainability

Read Now
Turning to Glitter in Management Studies – Why We Should Take ‘Unserious’ Glitter Serious to Understand New Management Practices
Business and Management INK
October 24, 2024

Turning to Glitter in Management Studies – Why We Should Take ‘Unserious’ Glitter Serious to Understand New Management Practices

Read Now
Utilizing Academic-Practitioner Partnering for Societal Impact

Utilizing Academic-Practitioner Partnering for Societal Impact

In this article, co-authors Natalie Slawinski, Bruna Brito, Jennifer Brenton, and Wendy Smith reflect on the inspiration behind their research article, “Reflections on deep academic–practitioner partnering for generative societal impact,” published in Strategic Organization.

Read Now
Trippin’ Forward: Management Research and the Development of Psychedelics

Trippin’ Forward: Management Research and the Development of Psychedelics

Charlie Smith reflects on his interest in psychedelic research, the topic of his research article, “Psychedelics, Psychedelic-Assisted Therapy and Employees’ Wellbeing,” published in Journal of Management Inquiry.

Read Now
Using Ethnography to Explore Entrepreneurial Extracurricular Activities

Using Ethnography to Explore Entrepreneurial Extracurricular Activities

Co-authors Birgitte Wraae and Nicolai Nybye reflect on the inspiration behind their research article, “Learning to Be “Me,” “the Team,” and “the Company” Through Entrepreneurial Extracurricular Activities: An Ethnographic Approach,” published in Entrepreneurship Education and Pedagogy.

Read Now
0 0 votes
Article Rating
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Newest
Oldest Most Voted
Inline Feedbacks
View all comments