Funking Up the Domain: How Outliers Skew Entrepreneurship Theory and Practice
“Our paper,” writes G. Christopher Crawford, “is both a cautionary tale for scholars before attempting replications and a roadmap for the domain to guide and encourage replication studies.” That dual warning and call to action permeates the research note that first authors Crawford, of the Rutgers Business School at Rutgers University, and Vitaliy Skorodziyevskiy of the Department of Management & Information Systems at Mississippi State University, with co-authors Casey J. Frid of the University of St. Thomas Opus College of Business, Thomas E. Nelson of the University of South Alabama, Zahra Booyavi of the Rutgers Business School, Diana M. Hechavarria of the Muma College of Business at the University of South Florida, Xuanye Li of the Rutgers Business School, Paul D. Reynolds of the Colorado-based Paul Reynolds and Associates, and Ehsan Teymourian of the Rutgers Business School, saw published recently in Entrepreneurship Theory and Practice. Here, Crawford talks about the origin of the underlying research, the trials in conducting it, and why the uphill effort that resulted in failure is a useful demonstration of how science happens.
Though the published title of our manuscript is “Advancing Entrepreneurship Theory Through Replication: A Case Study on Contemporary Methodological Challenges, Future Best Practices, and an Entreaty for Communality,” the title above was what we submitted with our original proposal to Entrepreneurship Theory and Practice’s special issue on knowledge accumulation in the domain (except we had a star (*) in place of funking’s “n”). The idea was built off of the Crawford et al. (2015) paper in the Journal of Business Venturing investigating PSED II, CAUSEE, KFS, and INC5000 (N~12,000) and the finding that all inputs and all outcomes (when measured on continuous scales) in entrepreneurship were distributed according to a power law. In these distributions, multiple high-performing outliers skew the distribution far to the right, making a traditional Gaussian normal curve look like a child’s slide, where the majority of observations are below the statistical mean.
We know that one outlier has the potential to influence the size and direction of effects, the significance of hypothesized relationships, and significantly alter the results of published works, but what happens when there are dozens of outliers in a sample?
Given that outliers can be found in human, social, intellectual, and financial capital, as well as expectations for future growth, entrepreneurial action, and environmental munificence, we proposed that these outliers were likely to have some influence on some of the domain’s seminal, and influential papers. Thus, we planned to conduct constructive replications of 30 seminal studies that used the same dataset(s) as did Crawford et al. (2015). Changes in the results of these papers could turn the domain on its head, we thought. In order to properly execute constructive replications, though, we first needed to exactly replicate the analyses (i.e., with the same N, same correlations, same coefficients), which would then give us a baseline for conducting additional experiments to test theory and identify boundary conditions. However, for the first few months of our investigation, the co-authors couldn’t correctly replicate any of the studies. For sure, it was a daunting task.
We forged ahead and kept trying. It wasn’t until after we hired eight statistical analysts (paying nearly US$38,000 in wages) and started asking authors of the papers for advice (about studies that were sometimes 20+ years old), data, and scripts that we were able to exactly replicate one study. And even after all that, we were only able to exactly replicate six of the 19 studies we attempted. With such a low success rate, though consistent with other domains, it becomes exceedingly difficult to build cumulative knowledge and research legitimacy. Therefore, instead of funking up the domain, we found that we first had to bring the domain together. Our paper is both a cautionary tale for scholars before attempting replications and a roadmap for the domain to guide and encourage replication studies.
And Miles to Go Before We Sleep
The real world is full of misinformation, exaggerations, bad intentions, and lies. That’s what drew me to science. To be sure, there were mistakes. My middle school textbook discussed the impending ice age, and the man-made hole in the ozone layer. Apparently, the Aqua Net favored by our theater department for turning normal hair into a plastic facsimile was also favored by hominids thousands of years ago. But over time, science comes closer and closer to the truth.
And those involved in the creation of knowledge try. We really do. But we get it wrong sometimes. I remember the first time I encountered this. I was devastated. At 14-years-old a lot of things devastate you. This instance was cold fusion. Now a footnote in history, at the time reasonable people all across the world thought that our energy problems were over. A team at some university or other had created a fusion reaction at room temperature. If this could be replicated then eventually every home in the world could have a tabletop reactor creating virtually unlimited energy, giving off only helium as a byproduct. But it couldn’t be replicated, and so, after several attempted replications, scientists moved on.
The point is, there was a process. It worked because of the free and accurate sharing of information. Years later, as a grad student I was sitting at a table with Benson Honig. We were chatting about data availability and replicating research. He had very strong opinions, opinions that I agreed with. I promised that I personally would provide my data to anyone who asked. (As a result of that promise I haven’t published out of my dissertation because I was robbed and my data disappeared. Such is life.) We agreed that data should be available, analysis should be spelled out, and everything should be made available, upon request, to other scientists who might want to extend the work or even just check it.
It was with that background I had the good fortune to hook up with a talented bunch of researcher/authors working on a large project using the PSED. This thing could be a dissertation or two, and will end up being several papers. The first was recently accepted. I’m thrilled that academe has found our work interesting enough to share. I’m disappointed in what we found.
We attempted first to reproduce a bunch of analysis from published papers, then replicate and extend the research. We assumed that the reproduction was going to be simple, and we’d start having problems when we attempted to replicate and extend. Nope. I was unsuccessful with every reproduction I attempted. I was unable to get to the same N as any of the studies I was examining. This was frustrating and I thought I was going to be kicked out of the club for my incompetence. Fortunately, others had similar problems and our failure became the story.
But our failure was different than the cold fusion failure. We didn’t fail the experiment; we couldn’t even set the experiment up. We couldn’t figure out from the published research what beakers, batteries, and solution we should be using to reproduce the experiment. This is a real problem, and one we describe in great detail in the paper, along with making several suggestions about how to limit this problem in the future.
What we discovered was, at least to us, interesting. That may be why it’s being published. But what we found wasn’t really new. Much has recently been written about problems with replication, first in psychology I believe, then in other fields. The term “replication crisis” was coined in the ‘early 2010’s’ according to everyone’s favorite secret research buddy, Wikipedia. Stories of faked data ending careers and invalidating research are not uncommon. Even completely tongue-in-cheek articles are shepherded through the publication process with journal editors missing the joke. There are problems. A fantastic part of the scientific process is becoming aware of these problems and working to fix them. Our article is an attempt to do part of that work and we hope that it makes a significant contribution.
