The Fuss Over P values

xkcd’s “real” interpretation of p-values

The American Statistical Association (ASA) has recently taken the unusual step of announcing a guideline document for preventing the misuse of p-values. They assert that scientists and policy-makers are using the p-value as a black-or-white decision parameter without truly understanding it and without inspecting the overall experiment / methodology / statistical framework. In this statement, the ASA advises researchers to refrain from drawing explicit scientific conclusions or making policy decisions based on just P values. They further advise that as part of scientific statistical analysis, the data analyses, statistical tests, and choices made in calculations should all be described in complete detail.

The ASA’s “statement on p-values: context, process, and purpose” can be accessed here, and the accompanying press release can be seen here.

I don’t want to go into the details of hypothesis testing and p-values here; those interested can take a course in statistical analysis or just scour google, or still, can take my graduate course on Data Analysis for the Earth Sciences.

To improve the conduct and interpretation of quantitative science, ASA has given the following six principles in the guideline document:

1. P-values can indicate how incompatible the data are with a specified statistical model.
2. P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.
3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.
4. Proper inference requires full reporting and transparency.
5. A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.
6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.

Many interesting and insightful news articles related to this released guideline have been published from different platforms. Nature News contains this directly linked article, and this detailed article from 2014 which discusses the pitfalls of relying too much on P-values. There are engaging  analyses presented in ScienceNews and Inside Higher Ed. RetractionWatch also published an interview with the ASA’s executive director.

I asked my colleague Dr. Asad Ali, an expert in statistical analysis, for his views;  he had this to say:

This p-value has become quite controversial in the last few years. People are abusing it intentionally or miss-using it because of lack of knowledge.

Rejection of a hypothesis does not always mean that it’s wrong, rather it can be also because our evidence (sampled data / observations) is not very representative of the underlying population.

Dr. Asad Ali works in the fields of Bayesian statistics, astrostatistics, and geostatistics, teaching extensive graduate-level courses on these topics at GREL. He is also a part of the eLisa mission data analysis team to observe gravitational waves from space.

To end this blog  post, I can’t resist sharing this brain-wrecking xkcd p-value joke:

A joke for statisticians only

Advertisements

Empirical Proof of the Central Limit Theorem in MATLAB

The Central Limit Theorem (CLT) is a fundamental theorem in probability and statistics which tells us that the sampling distribution of the mean is asymptotically Gaussian as long as the sample size is sufficiently large, no matter what distribution is followed by the population. The sampling distribution of the mean has a mean equal to the population mean (μ) and variance given by σ²/N, where σ² is the population variance and N is the sample size. Generally, the sample is considered sufficiently large for sample size greater than or equal to 30 (N ≥ 30). The variance of the sampling distribution of the mean is reduced by the factor N as the number of samples increases.

The ab initio proof of the CLT is rather complicated and requires strong knowledge of the underpinnings of probability theory¹. However, the CLT can be explored and understood empirically, through observations. Here is a MATLAB code I wrote to explore the CLT in a graduate class I am teaching on Data Analysis for the Earth Sciences.

MATLAB code for exploring the CLT

Population distribution (Rayleigh)

Sampling distribution of the mean with various sample size. Population distribution is Rayleigh.

¹Stark & Woods (2001) – Probability and Random Processes with Applications to Signal Processing (3rd Edition)