Statistical Significance and the PHC Curve

This book explains the importance of using the probability that the hypothesis is correct (PHC), an intuitive measure that anyone can understand, as an alternative to the p-value. In order to overcome the “reproducibility crisis” caused by the misuse of significance tests, this book provides a detailed explanation of the mechanism of p-hacking using significance tests, and concretely shows the merits of PHC as an alternative to p-values.

In March 2019, two impactful papers on statistics were published. One paper, "Moving to a World Beyond ‘p < 0.05’”, was featured in the scholarly journal The American Statistician, overseen by the American Statistical Association. The title of the first chapter is “Don't Say ‘Statistically Significant’”, and it uses the imperative form to clearly forbid the use of significance testing. Another paper, “Retire statistical significance”, was published in the prestigious scientific journal Nature. This commentary was endorsed by more than 800 scientists, advocating for the statement, “We agree, and call for the entire concept of statistical significance to be abandoned.”

Consider a study comparing the duration of hospital stays between treatments A and B. Previously, research conclusions were typically stated as: “There was a statistically significant difference at the 5% level in the average duration of hospital stays.” This phrasing is quite abstract. Instead, we present the following conclusion as an example: (1) The average duration of hospital stays for Group A is at least half a day shorter than for Group B. (2) 71% of patients in Group A have shorter hospital stays than the average for Group B. (3) Group A has an average hospital stay that is, on average, no more than 94% of that of Group B. Then, the probability that the expression is correct is shown. That is the PHC curve.