Statistical Significance vs Practical Importance: Four Worked Evidence Examples
A p-value does not measure effect size or practical importance. Interpret the estimate and confidence interval against a justified threshold for the relevant outcome, then consider harms, burden, bias, and applicability. Small p-values can accompany trivial effects, while larger p-values can leave important effects unresolved.
Ask how large the effect is before deciding whether it matters
Statistical evidence against a null value and practical importance are different questions. A small p-value does not measure the size of an effect. It also does not tell you whether the expected benefit is worth the burden, cost, or harms. The American Statistical Association's statement explicitly separates these questions and cautions against decisions based on a threshold alone.
For evidence writing, keep three pieces together: the effect estimate, its uncertainty, and a justified description of what would matter in this setting. If you have only 'p below 0.05', you do not yet have enough information to explain the result's practical meaning.
The worked examples below are fictional. They concern a comfort score from 0 to 10, measured one hour after an intervention, with higher scores better. We stipulate a +2-point between-group mean difference as an important-benefit threshold solely for this teaching exercise. It is not a validated clinical threshold, a within-person change threshold, or advice to adopt an intervention.
State the question and the threshold before reading the result
Write the outcome and comparison first: average comfort in the intervention group minus average comfort in the comparison group at one hour. A positive value favors the intervention on this score. That says nothing yet about adverse effects, later comfort, or how each individual responded.
A real importance threshold needs a defensible source for the relevant instrument, population, comparison, and use. A threshold for a meaningful change within a person may not be interchangeable with a threshold for an average difference between groups. Record that distinction rather than borrowing a familiar number from another study.
For this exercise, +2 points is supplied as a planning assumption so the arithmetic has a clear decision reference. If no defensible threshold were available in a real assignment, the correct response would be to describe the magnitude and uncertainty and explain that practical importance remains unresolved—not choose a cutoff that makes the result look favorable.
| Question | Specified answer | Boundary |
|---|---|---|
| What is measured? | Comfort score, 0–10, higher better | Not satisfaction, recovery, or long-term health |
| When? | One hour after intervention | No claim about later time points |
| Which comparison? | Intervention minus comparison-group mean | Positive means higher average comfort |
| Null value | 0 points | No mean difference on this outcome |
| Important-benefit threshold | +2 points, stipulated for this exercise | Between-group mean threshold; not a clinical standard |
| What else is needed for a decision? | Harms, burden, applicability, and evidence quality | The score alone does not determine adoption |
Work through four combinations of effect and precision
Assume each fictional analysis uses a two-sided normal approximation with the stated standard error. Its 95% confidence interval is estimate ± 1.96 × standard error, and the p-value tests a mean difference of zero under that same model. These assumptions make the examples reproducible; real analyses may require other methods.
Example A has an estimate of +0.30 and standard error 0.10. The interval is 0.30 ± 0.196, or +0.104 to +0.496. Its two-sided p-value is about 0.0027. The data are inconsistent with zero under the model, but the whole interval remains far below the stipulated +2-point important-benefit threshold.
Example B estimates +3.00 with standard error 2.00. Its interval is −0.92 to +6.92 and p is about 0.134. The estimate exceeds the threshold, but the interval also includes no difference and disadvantage. Calling this 'no benefit' would be as misleading as calling it established important benefit.
Example C also estimates +3.00 but with standard error 0.30. Its interval is +2.412 to +3.588 and p below 0.001. Under these assumptions, the interval lies above the teaching threshold. Example D estimates 0.00 with standard error 0.30: its interval is −0.588 to +0.588 and p = 1.00. D is compatible with small effects around zero and does not include effects as large as +2, but it does not prove the true effect is exactly zero.
| Example and estimate / SE | 95% interval and two-sided p-value | Relation to the stipulated +2 threshold |
|---|---|---|
| A: +0.30 / 0.10 | +0.104 to +0.496; p ≈ 0.0027 | Precisely below the important-benefit threshold |
| B: +3.00 / 2.00 | −0.92 to +6.92; p ≈ 0.134 | Includes effects below and above the threshold |
| C: +3.00 / 0.30 | +2.412 to +3.588; p below 0.001 | Entire interval above the threshold under the model |
| D: 0.00 / 0.30 | −0.588 to +0.588; p = 1.00 | Does not include +2; exact absence of effect is not proved |
Write a different conclusion for each result
A and C both have small p-values, but their magnitudes lead to different practical interpretations. B and D both have p-values above 0.05, yet B leaves a wide range unresolved while D is concentrated near zero. Grouping each pair under the same 'significant' or 'not significant' label hides the distinction the reader needs.
The model-based interpretations below are deliberately outcome-specific. They are not overall certainty judgments and they are not treatment recommendations. A biased estimate or a poorly applicable study can limit the use of even a narrow interval. Cochrane's interpretation guidance places effect magnitude and uncertainty within that wider appraisal.
For a real paper, quote neither the abstract's enthusiasm nor its dismissal as your own conclusion without checking the numerical result. A useful evidence paragraph explains which possibilities remain compatible with the data and why that matters for the stated question.
| Result | Avoid | More informative wording |
|---|---|---|
| A | The intervention produced an important improvement because p was small | The estimated mean improvement was small, and its interval remained below the stipulated important-benefit threshold |
| B | The intervention had no effect | The estimate favored the intervention, but the interval included disadvantage, no difference, and important benefit |
| C | The intervention should be adopted | The interval exceeded the stipulated threshold for this outcome; an adoption decision also requires appraisal of harms, burden, bias, and applicability |
| D | The two approaches are identical | The interval was concentrated near zero and excluded the stipulated +2-point benefit, without proving exact equality |
Check what could make the apparent result misleading
Look at the prespecified outcome and time point. If investigators measured many outcomes and highlighted only the smallest p-value, the selected result does not carry the same interpretation as a single planned comparison. Record the analysis context rather than treating the printed p-value as self-validating.
Check whether the effect is an absolute difference, relative measure, standardized difference, or adjusted estimate. A result of 0.3 on a standardized scale is not 0.3 points on the original questionnaire. Similarly, a relative reduction can appear large while the absolute change is small when the baseline event frequency is low.
Finally, separate the average from the distribution of individual outcomes. The mean difference in these examples cannot tell you that each participant improved by the same amount. A practical decision may also depend on who benefits, who experiences harm, how much effort the intervention requires, and whether the same conditions can be reproduced.
| Check | Record from the paper | Why the headline alone is insufficient |
|---|---|---|
| Outcome selection | Prespecified primary outcome and reported alternatives | A highlighted result may be selectively chosen |
| Scale and estimate | Units, range, direction, and adjusted status | Numbers on different scales are not interchangeable |
| Uncertainty | Interval, confidence level, and method | A point estimate alone conceals unresolved effects |
| Importance basis | Threshold source and intended interpretation | A convenient cutoff is not a validated decision rule |
| Harms and burden | What was measured, reported, or missing | A benefit on one outcome does not establish net benefit |
| Applicability | Population, setting, delivery, and follow-up | A result may not answer the target question |
Build a complete evidence paragraph from the numbers
For fictional example B, begin with the comparison and the estimate: 'At one hour, the estimated mean comfort score was 3.00 points higher in the intervention group on a 0–10 scale.' Add uncertainty: 'The 95% confidence interval ranged from 0.92 points lower to 6.92 points higher (two-sided p ≈ 0.134 under the stated normal model).'
Then explain the decision relevance: 'Although the estimate exceeded the exercise's stipulated +2-point threshold, the interval included no difference and disadvantage as well as important benefit. The study result therefore leaves important benefit unresolved; it should not be described as demonstrating no effect.' This tells the reader more than replacing the whole paragraph with 'not statistically significant.'
In a real review, finish with the relevant appraisal of bias, consistency, applicability, and the balance of outcomes. If the practical threshold has no adequate source, say so. The aim is to make the inference traceable from the study result to the sentence you write, with neither a small p-value nor a convenient threshold doing all the reasoning.
- Report the effect in meaningful units, with its interval and the relevant time point.
- Explain where the practical threshold came from and what kind of difference it represents.
- Describe unresolved benefit and disadvantage fairly when the interval allows both.
- Keep a statistical result, an evidence-certainty judgment, and a practical recommendation as separate steps.