Separate evidence, magnitude, and decision
| Question | Useful output | What it cannot decide alone |
|---|---|---|
| Is the result incompatible with the null model? | P-value and test specification | Magnitude or importance |
| How large is the estimated effect? | Raw effect and effect size | Whether uncertainty is acceptable |
| Which values remain compatible with the data? | Confidence interval | Context-specific consequences |
| Does the effect matter? | Field-defined meaningful difference, costs, and consequences | Statistical compatibility without uncertainty |
The ASA statement says that statistical significance does not measure effect size or importance and that scientific conclusions require fuller reporting.1 A star beside p < .05 cannot replace the estimate and interval.
Why sample size changes significance
A test statistic commonly scales an estimated effect by its standard error. With more information, the standard error can shrink, so a small effect may yield a small p-value. With little information, even a consequential estimate may have a wide interval and fail to cross a threshold.
This creates four possible combinations:
| Statistical result | Practical magnitude | Interpretation |
|---|---|---|
| Significant | Meaningful | Evidence and magnitude both support attention, subject to design and bias checks |
| Significant | Trivial | Precisely estimated but too small for the stated purpose |
| Not significant | Potentially meaningful | Evidence is inconclusive; interval may include meaningful and negligible values |
| Not significant | Negligible with narrow interval | Can support a negligible-effect conclusion only with justified bounds or an equivalence framework |
P-value interpretation explains why ordinary nonsignificance does not prove no effect.
Define “meaningful” before seeing the result
A minimum meaningful difference should come from the outcome scale, prior evidence, stakeholder consequences, measurement resolution, or a defensible decision model. There is no universal numeric threshold across outcomes.
Equivalence testing formalizes this idea with a smallest effect size of interest or equivalence bounds.3 The bounds need an external rationale. Choosing them after seeing the estimate merely to obtain a convenient conclusion defeats their purpose.
The excluded clinical variant is outside this page’s scope. ResultAtlas keeps the discussion methodological and does not provide patient-level or treatment interpretation.
Use raw and standardized effects together
Raw effects retain the outcome’s units and are often easiest to connect to consequences. Standardized effects support comparisons across scales but depend on the variability and estimator used. Good reporting can include both.
Lakens recommends calculating and reporting effect sizes for t-tests and ANOVAs to support cumulative interpretation rather than relying on significance labels.2 Cohen’s d interpretation shows why even an effect-size label needs attribution and context.
Read the confidence interval against meaningful bounds
Suppose an estimated difference is 1.8 units with a 95% interval from 0.2 to 3.4, and the context defines 3 units as the smallest meaningful magnitude.
- The interval excludes the null value 0 under the matching two-sided procedure.
- The point estimate is below the meaningful threshold.
- The interval includes effects both below and above that threshold.
Calling the result simply “significant” hides the practical uncertainty. The defensible conclusion is that the estimated effect is positive but the data do not sharply distinguish trivial from meaningful magnitudes under the stated criterion.
Practical significance is not permission to ignore validity
An important-looking estimate can still be biased by confounding, selection, measurement error, missing data, multiplicity, or model misspecification. Practical magnitude enters after the research design and estimand are understood; it does not repair weak evidence.
Likewise, a precise small effect may matter when it accumulates, affects many observations, or has low implementation cost. “Small” and “unimportant” are not synonyms.
Common reading errors
- Treating p < .05 as proof that an effect matters.
- Treating p > .05 as proof that the effect is zero.
- Choosing a meaningful-difference threshold after seeing the result.
- Reporting only a standardized label and hiding the original units.
- Ignoring the confidence interval around the effect.
- Assuming practical importance establishes causation.
A defensible reporting sentence
“The estimated difference was 1.8 units (95% CI [0.2, 3.4], p = .03). Although statistically significant under the prespecified rule, the interval spans the context-defined 3-unit minimum meaningful difference, so practical importance remains uncertain.”