International Debate

Replication Is More Common, But Still Too Rare International Debate
Message received, unfortunately.

Replication Is More Common, But Still Too Rare

August 14, 2014 1289

Do Not Replicate

Message received, unfortunately.

Earlier this month on Social Science Space, sociologist Robert DeVries argued “We Must Resist the Pressure to Be Interesting.” In discussing some of the problems dogging research-oriented journals, he addressed the processes that eventually weed out false results in the underlying disciplines:

Science is supposed to have a mechanism for correcting these sorts of errors. It is called replication, and it is one of the cornerstones of the scientific method. Someone else replicates what you did to see if they get the same results. Unfortunately, replication is another thing the science journals consider “boring” – so no one is doing it anymore. You can publish your tweaked and nudged and simplified results, safe in the knowledge that no-one will ever try exactly the same thing again and find something different.

While that might seem a little hyperbolic – surely replication, even if deprecated, isn’t deceased – but a new paper in Education Researcher suggests that, at least in that field, replication is on life support.

In “Facts Are More Important Than Novelty: Replication in the Education Sciences,” Duke’s Matthew C. Makel and the University of Connecticut’s Jonathan A. Plucker analyzed the complete publication history of the current 100 education journals with the highest five-year impact factor. They report that only 0.13 percent of published articles were replications (although that is still about four times the rate than the field saw a quarter century ago!).


VIDEO: Co-author Matthew C. Makel discusses key findings.


This is not good, “Although potentially beneficial for the individual researcher,” the authors write, “an over reliance on large effects from single studies drastically weakens the field as well as the likelihood of effective, evidence-based policy.”

Given that tiny rate, the good news, in a sense, is that when studies were replicated, more than two thirds – 67.4 percent – came up with the same findings as the original study, and only 13.1 percent failed to replicate any of the original findings.

Digging deeper, however, Makel and Plucker discovered that success (88.7 percent) was much more common when the replicators included researchers involved in the original research, and was dramatically less common (54 percent) when a completely new team took on the task. About half of all replications include one of the original authors, they found.

How does educational research stack up with other disciplines? A 2005 review of highly cited medical publications found that only 44 percent of replications produced results similar to the original study. A 2012 study by the same research team of the publication history of the top 100 psychology journals found that only 1.07 percent of publications were replications.

In an interview with Inside Higher Ed‘s Charlie Tyson, Makel found the lack of replication astounding. “When I talk to my friends in the natural sciences, they’re just baffled by how this is even a question or a controversy in psychology and education. Replication is such a normal part of the process for them.”

“The desire to distinguish ‘truth from nonsense’ is a constant struggle within science, and the education sciences are no exception,” Plucker was quoted by the American Educational Research Association (AERA), which publishes Educational Researcher. “The need to increase replication is apparent and permeates all levels of education research, to provide both confidence in our collective work and disincentives for misconduct.”

SAGE (which publishes the suite of AERA journals) and AERA are offering free access to this paper, which includes recommendation to increase the number and impact of replication studies, until the end of the month. Click here to view it.


Related Articles

Three Decades of Rural Health Research and a Bumper Crop of Insights from South Africa
Impact
March 27, 2024

Three Decades of Rural Health Research and a Bumper Crop of Insights from South Africa

Read Now
Using Translational Research as a Model for Long-Term Impact
Impact
March 21, 2024

Using Translational Research as a Model for Long-Term Impact

Read Now
Coping with Institutional Complexity and Voids: An Organization Design Perspective for Transnational Interorganizational Projects
Research
March 19, 2024

Coping with Institutional Complexity and Voids: An Organization Design Perspective for Transnational Interorganizational Projects

Read Now
The Importance of Using Proper Research Citations to Encourage Trustworthy News Reporting
Impact
February 26, 2024

The Importance of Using Proper Research Citations to Encourage Trustworthy News Reporting

Read Now
Revolutionizing Management Research with Immersive Research Methods

Revolutionizing Management Research with Immersive Research Methods

In this article, Anand van Zelderen, Nicky Dries, and Elise Marescaux reflect on their decision to explore nontraditional research.

Read Now
Why Don’t Algorithms Agree With Each Other?

Why Don’t Algorithms Agree With Each Other?

David Canter reviews his experience of filling in automated forms online for the same thing but getting very different answers, revealing the value systems built into these supposedly neutral processes.

Read Now
A Behavioral Scientist’s Take on the Dangers of Self-Censorship in Science

A Behavioral Scientist’s Take on the Dangers of Self-Censorship in Science

The word censorship might bring to mind authoritarian regimes, book-banning, and restrictions on a free press, but Cory Clark, a behavioral scientist at […]

Read Now
0 0 votes
Article Rating
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Inline Feedbacks
View all comments