The Tyranny of the P-Value
P-values don't tell us everything we need to know. Dr. Leslie Citrome explains why the P-value is not the same as an effect size — and why you need both to assess what a clinical trial actually found.
April 15, 2026
Leslie Citrome
Dr. Leslie Citrome, Section Editor of the ASCP Corner, explains why the P-value is not the same as an effect size — and why clinicians need both to assess how relevant a clinical trial finding could be in day-to-day practice.
Clinical Overview
How many times have you been shown a graph where the P-value is less than 0.05 and told the result is statistically significant — or less than 0.001, so it must be a very important outcome? P-values don’t tell us everything we need to know, and we suffer the tyranny of the P-value: we focus exclusively on the P-value and forget what it actually means.
The P-value is not an indication of how strong a comparison is in terms of effect size. It tells us the likelihood that any difference is not due to chance — whether we’re dealing with a true effect and not a fluke. Effect size tells us how big the treatment effect is in terms of reduction of symptoms or any other outcome of interest. The effect size is not the same as a P-value. In fact, you need both.
Relative effect sizes — relative risk, odds ratio, hazard ratio — can be confusing to interpret. A headline from the news not long ago read: “Aspirin Cuts Breast Cancer Risk.” The press reported that women who used aspirin at least once a week for six months reduced their risk of breast cancer by 20%. But the baseline risk of breast cancer in postmenopausal women who do not take aspirin regularly is 0.49%. A 20% relative reduction brings that to 0.41% — an absolute risk reduction of 0.08%. Put another way, a woman’s chance of being free from breast cancer over the next five years was 98.4% if she used aspirin and 98% if she did not. That’s a lot different than saying you’re reducing breast cancer risk by 20%.
Dr. Citrome favors absolute effect sizes, particularly Number Needed to Treat (NNT): how many patients need to be randomized to an experimental treatment versus placebo before expecting one additional outcome of interest. Small NNT values mean a bigger difference between what is being compared. If drug A results in remission 50% of the time and drug B results in remission 20% of the time, the NNT is 4 — meaning one additional remitted patient for every four patients when choosing A over B.
We have to use our brains when looking at effect sizes and personalize the treatment decision. Maybe drug A didn’t work for that individual patient in the past. Maybe it isn’t available on the formulary. Maybe it’s associated with a side effect the patient absolutely wants to avoid. The first step is interpreting that clinical trial data and looking to see how relevant it could be in day-to-day practice.
Key Takeaways
- The P-value tells us the likelihood that any difference is not due to chance — not how large the treatment effect is.
- Effect size and statistical significance are not the same thing — you need both.
- Relative measures like "20% risk reduction" can be confusing to interpret; absolute measures like Number Needed to Treat give a more direct picture.
- Number Needed to Treat tells you how many patients need to be randomized to an experimental treatment versus placebo before expecting one additional outcome of interest — smaller values mean a bigger difference.
- Effect sizes must be interpreted alongside individual patient circumstances — prior response, formulary access, and side effect tolerability all shape the final decision.
P-values don't tell us everything we need to know, and we suffer the tyranny of the P-value, as I call it. We focus exclusively on the P-value and forget what it actually means.
From Psychiatrist.com
Stay Current
We send one focused email per month. No spam — unsubscribe anytime.
