Research

Unknown Unknowns: The War on Null and Negative Results Research
Many times a negative can be very enlightening.

Unknown Unknowns: The War on Null and Negative Results

September 19, 2014 1643

X-ray

Many times a negative can be very enlightening.

There is a lot of pressure on academic researchers. Rightly or wrongly, it is widely held that researchers must publish prodigiously and secure funding consistently to advance their careers and that their research outputs must achieve high ‘impact’ to be of value.

An unintended consequence of this ‘impact culture’ may be a publication bias against research findings whose implications are not immediately obvious, or which may be deemed less novel, interesting, or eye-catching. Some research suggests that this bias may extend to research that yields negative and/or null findings.

Wellcome Trust logo

This post by Jonathon Kram and Adam Dinsmore originally appeared on the Wellcome Trust blog under the title, “Unknown unknowns: Is there a selection bias against null results?” It is reposted here under a Creative Commons Attribution-Non-Commercial 2.0 UK: England & Wales licence.

There are a few definitions of ‘negative finding’ currently in use, but the idea of significance is central to the concept. A negative result fails to reject the null hypothesis –the proposition that no significant effect is present – within the bounds of an experiment and to a level of statistical rigor. This can occur when a treatment is found to have no greater efficacy than placebo, or when no difference is found between two distinct groups of people (e.g. conservatives and liberals) on some variable of interest.

That said, the term is also used to refer to findings of slightly different but related kinds, including:

  • Incremental improvements: Minor refinements to previously reported hypotheses (e.g. where a previous result can only be replicated in certain conditions).
  • Refutations: Results that defy expectation, whether that’s the previous scientific consensus or a researcher’s original intent.

A bias against sharing statistically insignificant results (alongside incremental refinements and refutations) could have detrimental effects to the progression of science and our knowledge. Researchers could waste time pursuing hypotheses which have already been disconfirmed. Clinicians could make treatment decisions based on incomplete evidence. The time and energy of human participants could be wasted in research studies which were never subsequently reported.

Worryingly, there is some evidence that this is already happening.

Hopewell et al (2007) found that research papers which reported statistically significant results were approximately four times as likely to be published as their negative counterparts. The work of Daniele Fanelli (2012) suggests that the proportion of papers publishing negative findings may be decreasing over time; in 2007 just 14 percent of papers indexed by ISI which declared to have tested a hypothesis reported a negative finding, down from 30 percent in 1990. Recently Franco et al (2014) found that negative findings are less likely to be written up for publication than more positive results.

Several attempts have been made to encourage the publication of negative findings. BioMed Central’s Journal for Negative Results in BioMedicine has published articles which either report negative findings or discuss their place in science since 2002, while Open Access publishers PLOS One and PeerJ encourage the submission of any scientific results including negative findings.

Away from conventional publication the online datahub FigShare actively encourages the upload of datasets containing null and negative findings, without requiring that the researchers responsible for them prepare a full manuscript for submission.

As a research funder, we are keen that all research findings be accessible and presented to the world; including those that describe a breakthrough, disprove a theory or demonstrate a null result. Everyone in research has an opinion of what makes “good science”, from the researchers through to the readers of a paper. Researchers, journal editors and peer-reviewers all make value judgments about what’s worth telling the rest of the world about. However, if there are processes in play that inadvertently (or advertently) discriminate against the availability of a certain type of result and its data, we risk needless repetition of research and, potentially worse, proliferation of apparent knowledge that may be wrong.

Through the open access movement and the potential of digital technologies, we have the opportunity to enable accessibility to all the findings of research and reduce research waste. If the studies cited above accurately reflect a bias against negative findings in the biomedical literature, then it is the duty of all stakeholders in the delivery of science and knowledge to engage with the issue and explore ways of correcting it.

There is no benchmark to measure the progress of science against but there are hints that there may be a lot of wasted effort caused by our attitude towards negative findings and their unloved siblings. It is only by building a stronger evidence base around negative findings, and exploring the factors that influence publication bias, that we can explore the severity of the problem and make steps towards improving the health of science.

***

The Wellcome Trust policy position on research involving human participants states that negative results and full disclosure are expected. We are committed to maximising the availability of and value of research data.


Jonathon Kram is a research assistant, and Adam Dinsmore an evaluation officer, working within the Wellcome Trust’s Strategic Planning & Policy Unit. The unit provides analysis, research and advice on policy and evaluation issues relevant to the trust’s mission to achieve extraordinary improvements in human and animal health by supporting the brightest minds in biomedical research.

View all posts by Jonathon Kram and Adam Dinsmore

Related Articles

Exploring the ‘Publish or Perish’ Mentality and its Impact on Research Paper Retractions
Research
October 10, 2024

Exploring the ‘Publish or Perish’ Mentality and its Impact on Research Paper Retractions

Read Now
Lee Miller: Ethics, photography and ethnography
News
September 30, 2024

Lee Miller: Ethics, photography and ethnography

Read Now
NSF Seeks Input on Research Ethics
Ethics
September 11, 2024

NSF Seeks Input on Research Ethics

Read Now
Megan Stevenson on Why Interventions in the Criminal Justice System Don’t Work
Social Science Bites
July 1, 2024

Megan Stevenson on Why Interventions in the Criminal Justice System Don’t Work

Read Now
How ‘Dad Jokes’ Help Children Learn How To Handle Embarrassment

How ‘Dad Jokes’ Help Children Learn How To Handle Embarrassment

Yes, dad jokes can be fun. They play an important role in how we interact with our kids. But dad jokes may also help prepare them to handle embarrassment later in life.

Read Now
How Social Science Can Hurt Those It Loves

How Social Science Can Hurt Those It Loves

David Canter rues the way psychologists and other social scientists too often emasculate important questions by forcing them into the straitjacket of limited scientific methods.

Read Now
Digital Scholarly Records are Facing New Risks

Digital Scholarly Records are Facing New Risks

Drawing on a study of Crossref DOI data, Martin Eve finds evidence to suggest that the current standard of digital preservation could fall worryingly short of ensuring persistent accurate record of scholarly works.

Read Now
0 0 votes
Article Rating
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Newest
Oldest Most Voted
Inline Feedbacks
View all comments