Replication Is More Common, But Still Too Rare
Message received, unfortunately.
Earlier this month on Social Science Space, sociologist Robert DeVries argued “We Must Resist the Pressure to Be Interesting.” In discussing some of the problems dogging research-oriented journals, he addressed the processes that eventually weed out false results in the underlying disciplines:
Science is supposed to have a mechanism for correcting these sorts of errors. It is called replication, and it is one of the cornerstones of the scientific method. Someone else replicates what you did to see if they get the same results. Unfortunately, replication is another thing the science journals consider “boring” – so no one is doing it anymore. You can publish your tweaked and nudged and simplified results, safe in the knowledge that no-one will ever try exactly the same thing again and find something different.
While that might seem a little hyperbolic – surely replication, even if deprecated, isn’t deceased – but a new paper in Education Researcher suggests that, at least in that field, replication is on life support.
In “Facts Are More Important Than Novelty: Replication in the Education Sciences,” Duke’s Matthew C. Makel and the University of Connecticut’s Jonathan A. Plucker analyzed the complete publication history of the current 100 education journals with the highest five-year impact factor. They report that only 0.13 percent of published articles were replications (although that is still about four times the rate than the field saw a quarter century ago!).
VIDEO: Co-author Matthew C. Makel discusses key findings.
This is not good, “Although potentially beneficial for the individual researcher,” the authors write, “an over reliance on large effects from single studies drastically weakens the field as well as the likelihood of effective, evidence-based policy.”
Given that tiny rate, the good news, in a sense, is that when studies were replicated, more than two thirds – 67.4 percent – came up with the same findings as the original study, and only 13.1 percent failed to replicate any of the original findings.
Digging deeper, however, Makel and Plucker discovered that success (88.7 percent) was much more common when the replicators included researchers involved in the original research, and was dramatically less common (54 percent) when a completely new team took on the task. About half of all replications include one of the original authors, they found.
How does educational research stack up with other disciplines? A 2005 review of highly cited medical publications found that only 44 percent of replications produced results similar to the original study. A 2012 study by the same research team of the publication history of the top 100 psychology journals found that only 1.07 percent of publications were replications.
In an interview with Inside Higher Ed‘s Charlie Tyson, Makel found the lack of replication astounding. “When I talk to my friends in the natural sciences, they’re just baffled by how this is even a question or a controversy in psychology and education. Replication is such a normal part of the process for them.”
“The desire to distinguish ‘truth from nonsense’ is a constant struggle within science, and the education sciences are no exception,” Plucker was quoted by the American Educational Research Association (AERA), which publishes Educational Researcher. “The need to increase replication is apparent and permeates all levels of education research, to provide both confidence in our collective work and disincentives for misconduct.”
SAGE (which publishes the suite of AERA journals) and AERA are offering free access to this paper, which includes recommendation to increase the number and impact of replication studies, until the end of the month. Click here to view it.
