Statistical Association Takes on Use, Abuse of P-values

Even as it insists it’s not really saying anything new, the American Statistical Association Board of Directors has laid down a marker in the debate over what constitutes “statistical significance.”

After months of discussion with a blue-chip panel of experts, the association has laid down six principles about what constitutes good practice when dealing with p-values, which it informally defines as “the probability under a specified statistical model that a statistical summary of the data (for example, the sample mean difference between two compared groups) would be equal to or more extreme than its observed value.” It has been common shorthand to use P < 0.05 as meaning a result is significant. As physiologist Lewis Halsey explained in these pages, “Although P is never proof that there is a difference – scientific studies never prove things, they only provide a degree of evidence for them – studies with low P values are thought to be convincing, and so are not often repeated to be sure the results are correct.”

As Halsey further explained, this has proved an increasingly fraught decision with at least one journal banning the use of p-values in their accepted papers and debates over the replicability of research poking holes in using p-values and other ‘bright-line’ tests.

The principles include statements such as “Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold” and “Proper inference requires full reporting and transparency,” statements that are both self-evident and yet striking at the same time.

Today’s statement is the first time the ASA – which describes itself as the world’s largest community of statisticians – has taken a position on a specific matter of statistical practice. “This statement does not seek to resolve all the issues relating to sound statistical practice, nor to settle foundational controversies,” the ASA statement reads. “Rather, the statement articulates in non-technical terms a few select principles that could improve the conduct or interpretation of quantitative science, according to widespread consensus in the statistical community.”

A release from the ASA to the statistics community does an admirable job of summing up what is already a concise statement (the full statement is available for free at The American Statistician website):

“Widespread use of ‘statistical significance’ (generally interpreted as ‘p < 0.05′) as a license for making a claim of a scientific finding (or implied truth) leads to considerable distortion of the scientific process,” says the ASA statement (in part). By putting the authority of the world’s largest community of statisticians behind such a statement, we seek to begin a broad-based discussion of how to more effectively and appropriately use statistical methods as part of the scientific reasoning process.

In short, we envision a new era, in which the broad scientific community recognizes what statisticians have been advocating for many years. In this “post p < .05 era,” the full power of statistical argumentation in all its nuance will be brought to bear to advance science, rather than making decisions simply by reducing complex models and methods to a single number and its relationship to an arbitrary threshold. This new era would be marked by radical change to how editorial decisions are made regarding what is publishable, removing the temptation to inappropriately hunt for statistical significance as a justification for publication. In such an era, every aspect of the investigative process would have its appropriate weight in the ultimate decision about the value of a research contribution.

The message from the ASA included a quick origin story by Ronald L. Wasserstein and Nicole A. Lazar about the statement, demonstrating the seriousness of the concern and the complexity of saying something that was, in itself, significant about the issue. “Though there was disagreement on exactly what the statement should say,” the message on ‘context, process, and purpose’ explained, “there was high agreement that the ASA should be speaking out about these matters.

“Let’s be clear. Nothing in the ASA statement is new. Statisticians and others have been sounding the alarm about these matters for decades, to little avail. We hoped that a statement from the world’s largest professional association of statisticians would open a fresh discussion and draw renewed and vigorous attention to changing the practice of science with regards to the use of statistical inference.”