Publication Bias in the Organizational Sciences
Meta-analysis has emerged as an important means of gathering cumulative scientific data, but if the results are skewed, it can hinder rather than help knowledge advancement in the field. Dr. Sven Kepes of Virginia Commonwealth University co-authored “Publication Bias in the Organizational Sciences,” published on July 31, 2012 in Organizational Research Methods, with George C. Banks and Michael McDaniel, both of Virginia Commonwealth University, and Deborah L. Whetzel of Human Resources Research Organization (HumRRO). Dr. Kepes kindly summarized the article, which argues that organizational science researchers in particular must conduct rigorous assessments in order to prevent publication bias:
Publication bias poses multiple threats to the accuracy of meta-analytically derived effect sizes and related statistics (Rothstein, Sutton, & Borenstein, 2005a). Publication bias occurs when the publicaly available literature is not representative of all studies on the relationship of interest (Rothstein et al., 2005a). This bias stems from the tendency to submit and/or publish studies with statistically significant results, rather than basing submission and publication decisions on the quality of the research (Greenwald, 1975; Orlitzky, in press; Porter, 1992; Rothstein, Sutton, & Borenstein, 2005b). As a result, meta-analytic reviews may overestimate the mean effect.
A review of the literature indicates that, unlike meta-analytic reviews in medicine, research in the organizational sciences (e.g., Management and Industrial/Organizational Psychology) tends to pay little attention to this issue. In this paper, we introduce advances in meta-analytic techniques from the medical and related sciences for a comprehensive assessment and evaluation of publication bias. We illustrate their use on a dataset on employment interview validities (McDaniel, Whetzel, Schmidt, & Maurer, 1994). Using multiple methods, including contour-enhanced funnel plots, trim and fill, Egger’s test of the intercept, Begg and Mazumdar’s rank correlation, meta-regression, cumulative meta-analysis, and selection models, we find limited evidence of publication bias in McDaniel et al.’s (1994) data. However, we do find that effect sizes from journal articles tend to be larger in magnitude than effect sizes from other sources. Specifically, meta-analytic mean estimates derived from effect size distributions from journal articles tend to overestimate the effect size (e.g., meta-analytic mean estimates for structured interviews: all data sources: r ̅_o=.27; journal data: r ̅_o=.34; non-journal data r ̅_o=.19).
Our results support findings from other literature streams in the social (e.g., Banks, Kepes, & Banks, in press; Banks, Kepes, & McDaniel, 2012; McDaniel, Rothstein, & Whetzel, 2006; Renkewitz, Fuchs, & Fiedler, 2011) and medical (e.g., Chalmers, 1990; Dickersin, 2005; Song et al., 2010; Sutton, 2005) sciences. Our findings thus suggest that conclusions from meta-analytic reviews that use predominantly journal articles as their data source could be more erroneous than meta-analytic results based on data sets that include many non-journal sources. This is troublesome because many researchers believe that research published in journal articles is the most accurate available. Together with prior research in the organizational sciences on publication bias (e.g., Banks, Kepes, & McDaniel, 2012; McDaniel et al., 2006), we can thus conclude that Dalton et al.’s (2011) assertion that publication bias does not pose a threat to the accuracy of meta-analytic results in the I/O Psychology and Management literatures is likely to be erroneous. Aligned with the Meta-Analysis Reporting Standards (MARS) from the American Psychology Association (2008, 2010), we strongly recommend that all future meta-analytic reviews assess the issue of publication bias empirically with appropriate methods.
View references here.
Click here to read the entire article in Organizational Research Methods, and follow this link to learn more about the journal. Interested in receiving email alerts about newly published issues and articles like these? Then click here!
