The American Statistical Association (ASA) has recently taken the unusual step of announcing a guideline document for preventing the misuse of p-values. They assert that scientists and policy-makers are using the p-value as a black-or-white decision parameter without truly understanding it and without inspecting the overall experiment / methodology / statistical framework. In this statement, the ASA advises researchers to refrain from drawing explicit scientific conclusions or making policy decisions based on just P values. They further advise that as part of scientific statistical analysis, the data analyses, statistical tests, and choices made in calculations should all be described in complete detail.
The ASA’s “statement on p-values: context, process, and purpose” can be accessed here, and the accompanying press release can be seen here.
I don’t want to go into the details of hypothesis testing and p-values here; those interested can take a course in statistical analysis or just scour google, or still, can take my graduate course on Data Analysis for the Earth Sciences.
To improve the conduct and interpretation of quantitative science, ASA has given the following six principles in the guideline document:
- P-values can indicate how incompatible the data are with a specified statistical model.
- P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.
- Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.
- Proper inference requires full reporting and transparency.
- A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.
- By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.
Many interesting and insightful news articles related to this released guideline have been published from different platforms. Nature News contains this directly linked article, and this detailed article from 2014 which discusses the pitfalls of relying too much on P-values. There are engaging analyses presented in ScienceNews and Inside Higher Ed. RetractionWatch also published an interview with the ASA’s executive director.
I asked my colleague Dr. Asad Ali, an expert in statistical analysis, for his views; he had this to say:
This p-value has become quite controversial in the last few years. People are abusing it intentionally or miss-using it because of lack of knowledge.
Rejection of a hypothesis does not always mean that it’s wrong, rather it can be also because our evidence (sampled data / observations) is not very representative of the underlying population.