A common mistake in data analysis is treating statistical significance as proof that a result is important.

Suppose an experiment finds that a new website design increases conversion rates, and the statistical test produces:

p-value = 0.003

It may be tempting to conclude:

The new design has a meaningful impact.

But statistical significance alone doesn’t tell you how large the impact is.

The actual difference might be tiny.

For example:

Old conversion rate: 10.00%
New conversion rate: 10.05%

With a sufficiently large sample, that small difference could be statistically significant.

This is where effect size becomes important.

Effect size tells you how large a difference or relationship is, while statistical significance helps determine whether the observed evidence is inconsistent with a specified null hypothesis.

The two concepts answer different questions.

Statistical significance asks whether the observed data provide sufficient evidence against a null hypothesis under a chosen statistical model and significance threshold.

Effect size asks how large the observed difference or relationship is.

In simplified terms:

Statistical significance
        ↓
"Is there evidence of a difference?"

Effect size
        ↓
"How large is the difference?"

A strong analysis often considers both.

Statistical significance
        +
Effect size
        +
Confidence interval
        +
Practical context

Together, these provide a much better interpretation than a p-value alone.

What Is Statistical Significance?

Statistical significance is typically evaluated using a hypothesis test.

Suppose you want to determine whether a new marketing campaign changes the average purchase value.

You might define:

Null hypothesis (H₀):
No difference exists.

Alternative hypothesis (H₁):
A difference exists.

You then calculate a test statistic and p-value.

If the p-value is below your chosen significance level, often:

α = 0.05

you may reject the null hypothesis.

For example:

p = 0.02
α = 0.05

Because:

0.02 < 0.05

the result is statistically significant under that decision rule.

But this does not tell you whether the effect is large.

What Is Effect Size?

Effect size quantifies the magnitude of an observed difference or relationship.

For example, suppose two groups have average test scores of:

Group A = 80
Group B = 85

The difference is:

85 - 80 = 5 points

That tells you the raw difference.

But whether five points is large depends on the variability and context.

Effect size provides standardized or otherwise interpretable ways of describing magnitude.

Common effect-size measures include:

• Cohen’s d
• Hedges’ g
• Pearson’s correlation
• Odds ratio
• Risk ratio
• Risk difference
• Eta squared
• Partial eta squared

The appropriate measure depends on the analysis.

Why P-Values Don’t Measure Effect Size

A p-value does not tell you:

How large is the effect?

It describes how compatible the observed data are with the null hypothesis under the assumptions of the statistical test.

A very small p-value can occur with a tiny effect if the sample is large enough.

Conversely, a relatively large effect may fail to reach conventional statistical significance if the sample is small or the data are highly variable.

This is one of the most important ideas in statistical interpretation.

A Simple Example

Imagine two groups:

Group A: 100,000 people
Group B: 100,000 people

Suppose the average outcome differs by only:

0.2 units

Because the sample is enormous, the statistical test may produce:

p < 0.001

The difference is statistically significant.

But whether a 0.2-unit difference matters depends on the application.

Now imagine a second study:

Group A: 20 people
Group B: 20 people

and the observed difference is:

10 units

This may represent a much larger practical effect, yet the result might not achieve statistical significance because the sample is small.

Therefore:

Small effect + large sample
→ Can be statistically significant

Large effect + small sample
→ May not be statistically significant

Statistical Significance vs Practical Significance

This distinction is critical.

Statistical significance concerns evidence under a statistical model.

Practical significance asks whether the size of the effect matters in the real world.

Suppose a company changes its checkout process.

Before:

Conversion = 20.00%

After:

Conversion = 20.02%

With millions of visitors, this difference might be statistically significant.

But if implementation costs millions of dollars, the business may decide the improvement isn’t worthwhile.

The effect is statistically detectable but potentially not practically important.

Effect Size Makes Magnitude Visible

Consider two experiments.

Experiment A

Difference = 0.2%
p-value = 0.001

Experiment B

Difference = 8%
p-value = 0.08

Experiment A is statistically significant at the 5% level.

Experiment B is not.

But Experiment B may have greater practical importance if the estimated 8% effect is credible and would justify further investigation.

The p-value alone doesn’t communicate that distinction.

Common Effect Size: Cohen’s d

Cohen’s d is frequently used to measure the standardized difference between two means.

The basic idea is:

Difference between means
------------------------
      Standard deviation

A larger absolute value of Cohen’s d indicates a larger standardized difference.

where:

• $\bar{x}_1$ is the mean of group 1
• $\bar{x}_2$ is the mean of group 2
• $s_{pooled}$ is the pooled standard deviation

Interpreting Cohen’s d

A commonly used rough guideline is:

Cohen’s d	Approximate interpretation
0.2	Small
0.5	Medium
0.8	Large

These are rules of thumb, not universal thresholds.

An effect size of 0.2 could be extremely important in one domain and practically irrelevant in another.

Context always matters.

Example of Cohen’s d

Suppose:

Group A mean = 70
Group B mean = 75
Pooled SD = 10

Then:

d = (75 - 70) / 10
  = 0.50

The standardized difference is approximately 0.5, often described as a medium effect using conventional guidelines.

But the real-world interpretation depends on what the measurement represents.

Correlation as Effect Size

When analyzing the relationship between two continuous variables, Pearson’s correlation coefficient can serve as an effect-size measure.

It ranges from:

-1 to +1

For example:

r = 0.80

indicates a strong positive linear association.

While:

r = 0.05

indicates a very weak linear association.

Importantly, correlation does not establish causation.

Odds Ratios

For binary outcomes, odds ratios are commonly used.

Suppose you are studying whether exposure is associated with an outcome.

An odds ratio of:

OR = 1

indicates no difference in odds.

An odds ratio above 1 indicates higher odds in the exposed group, while an odds ratio below 1 indicates lower odds.

The magnitude should be interpreted in the context of the underlying event rate.

Risk Difference

Risk difference is often easier to interpret than an odds ratio.

Suppose:

Treatment risk = 8%
Control risk = 12%

Then:

Risk difference = 8% - 12%
                = -4 percentage points

The treatment is associated with a four-percentage-point lower risk.

This can be much easier for a general audience to understand than an odds ratio.

Relative Change vs Absolute Change

Another important distinction is between relative and absolute effects.

Suppose conversion increases from:

10% → 12%

The absolute change is:

+2 percentage points

The relative change is:

(12 - 10) / 10 = 20%

Both statements are mathematically correct.

But they communicate different things.

This is why effect-size reporting should be clear about the scale being used.

Confidence Intervals Add Important Context

Effect sizes are estimates, not exact truths.

A confidence interval provides information about the uncertainty surrounding an estimate.

Suppose:

Effect size = 0.50
95% CI = [0.30, 0.70]

This communicates more information than:

p = 0.01

alone.

You can see:

• The estimated magnitude
• The uncertainty
• The range of values compatible with the chosen confidence procedure

Statistical Significance and Confidence Intervals

For many common hypothesis tests, a confidence interval can help show whether values corresponding to the null effect are included.

For example, suppose the estimated mean difference is:

5

with:

95% CI = [2, 8]

The interval excludes zero, which is consistent with rejecting a zero-difference null hypothesis in a corresponding two-sided test at the 5% level.

But remember that confidence intervals and p-values answer somewhat different questions and rely on the assumptions of the underlying method.

Sample Size Changes the Picture

Sample size has a major influence on statistical significance.

A simplified relationship is:

Larger sample
      ↓
Smaller standard error
      ↓
Greater precision
      ↓
Smaller effects can become detectable

This is why large datasets can produce very small p-values for effects that may have limited practical importance.

It is not a flaw in statistical testing.

It is a reason to interpret statistical significance alongside effect size and context.

Statistical Power

The opposite problem can occur with small samples.

Suppose a real effect exists, but the study contains too few observations.

The test may produce:

p = 0.12

even though the estimated effect is meaningful.

This can happen because the estimate is imprecise.

This is related to statistical power.

Power is influenced by factors such as:

• Effect size
• Sample size
• Variability
• Significance level
• Study design

A non-significant result therefore does not automatically prove that no meaningful effect exists.

Type I and Type II Errors

Hypothesis testing involves uncertainty.

A Type I error occurs when a true null hypothesis is rejected.

A Type II error occurs when a false null hypothesis is not rejected.

These concepts help explain why statistical significance should not be interpreted as certainty.

Statistical test
      ↓
Evidence assessment
      ↓
Decision under uncertainty

It does not produce a guarantee that an effect is real or important.

Statistical Significance Does Not Mean “Important”

Consider:

Effect = 0.01 units
p < 0.001

This result can be statistically significant.

But whether 0.01 units matters depends entirely on the domain.

For a manufacturing process where 0.01 units represents a major defect reduction, it could matter greatly.

For another application, it could be irrelevant.

Statistical significance is therefore not a synonym for importance.

A Better Reporting Style

Instead of writing:

The treatment was statistically significant (p < 0.05).

A stronger report might say:

The treatment was associated with a 5-point increase in the outcome, with a 95% confidence interval of 2 to 8 points. The result was statistically significant under the prespecified test.

This communicates:

• Direction
• Magnitude
• Uncertainty
• Statistical evidence

Reporting Effect Size in Python

Python’s scientific ecosystem provides several ways to calculate effect sizes.

For a simple standardized mean difference, you can calculate Cohen’s d directly:

import numpy as np

group_a = np.array([70, 72, 68, 75, 71])
group_b = np.array([76, 79, 74, 77, 80])

mean_difference = group_b.mean() - group_a.mean()

pooled_sd = np.sqrt(
    (
        group_a.var(ddof=1) +
        group_b.var(ddof=1)
    ) / 2
)

cohens_d = mean_difference / pooled_sd

print("Mean difference:", mean_difference)
print("Cohen's d:", cohens_d)

For serious analysis, use an effect-size implementation appropriate to your study design rather than relying on a simplified formula when assumptions differ.

Testing Statistical Significance in Python

For example, an independent two-sample t-test can be performed using SciPy:

from scipy.stats import ttest_ind

result = ttest_ind(
    group_a,
    group_b
)

print("t-statistic:", result.statistic)
print("p-value:", result.pvalue)

You can then report both the statistical test and effect size.

For example:

Mean difference: ...
Cohen's d: ...
p-value: ...

The exact test should match the research design and assumptions.

Effect Size for A/B Testing

Effect size is particularly useful in A/B testing.

Suppose:

Control:
10,000 visitors
500 conversions

Treatment:
10,000 visitors
550 conversions

The conversion rates are:

Control = 5.0%
Treatment = 5.5%

The absolute improvement is:

0.5 percentage points

The relative improvement is:

10%

A statistical test can determine whether the observed difference provides sufficient evidence against the null hypothesis.

But the business question remains:

Is a 0.5-percentage-point improvement worth implementing?

That requires effect size and practical context.

Effect Size in Machine Learning

Effect size also has applications beyond classical experiments.

For example, you might compare:

• Model performance
• Feature distributions
• Treatment groups
• Error rates
• Conversion rates

Suppose:

Model A accuracy = 91.2%
Model B accuracy = 91.4%

The difference is:

0.2 percentage points

Whether that improvement is meaningful depends on:

• Sample size
• Variability
• Cost
• Latency
• Model complexity
• Business impact

A statistically significant improvement isn’t automatically worth deploying a more expensive model.

Effect Size vs Statistical Significance

Question	Statistical Significance	Effect Size
Main question	Is there evidence against the null?	How large is the effect?
Depends on sample size?	Strongly	Less directly
Measures magnitude?	No	Yes
Uses p-value?	Yes	Not inherently
Helps assess practical importance?	Limited	Yes
Useful with confidence intervals?	Yes	Yes
Should be reported alone?	Usually not	Usually not

The two measures complement each other.

Four Possible Outcomes

When analyzing results, you can encounter four broad situations.

1. Significant + Large Effect

p < 0.05
Large effect

This is often compelling evidence of a potentially important difference, assuming the study design and assumptions are appropriate.

2. Significant + Small Effect

p < 0.05
Small effect

The difference may be statistically detectable but practically unimportant.

Large datasets commonly produce this situation.

3. Non-Significant + Large Estimated Effect

p > 0.05
Large estimated effect

The study may have insufficient precision or power.

The confidence interval should be examined carefully.

4. Non-Significant + Small Effect

p > 0.05
Small estimated effect

This may provide less evidence of a meaningful effect, but uncertainty and the confidence interval still matter.

Don’t Use Arbitrary Effect-Size Thresholds

Rules such as:

d = 0.2 → small
d = 0.5 → medium
d = 0.8 → large

can be useful as rough reference points.

But they shouldn’t replace domain knowledge.

For example, a tiny improvement in a high-volume system may generate substantial financial value.

Likewise, a large standardized effect may have little operational value if the outcome itself isn’t important.

The Role of Domain Knowledge

Statistical analysis should ultimately answer a real-world question.

Suppose a medical intervention reduces a risk from:

2.0% → 1.5%

That’s a:

0.5 percentage-point

absolute reduction.

Whether that matters depends on:

• Severity of the outcome
• Treatment cost
• Side effects
• Number of people affected
• Alternative treatments

Statistics can quantify the evidence.

Domain knowledge determines whether the effect is meaningful.

A Better Statistical Workflow

Instead of:

Run test
 ↓
Check p-value
 ↓
Declare success

use:

Define research question
        ↓
Choose appropriate analysis
        ↓
Estimate effect
        ↓
Calculate uncertainty
        ↓
Assess statistical evidence
        ↓
Interpret practical importance
        ↓
Make decision

This produces much more informative conclusions.

What Should You Report?

For most comparative analyses, consider reporting:

1. Group estimates

Group A = 80
Group B = 85

2. Absolute difference

Difference = 5

3. Effect size

Cohen's d = ...

4. Confidence interval

95% CI = [...]

5. Statistical test

p = ...

6. Practical interpretation

Explain whether the magnitude matters in the real-world context.

A Practical Interpretation Template

A useful reporting structure is:

The estimated effect was X units
(95% CI: A to B).

The corresponding effect size was Y.

The statistical test produced
p = Z.

The observed effect is [likely/unlikely]
to be practically important given [domain context].

This is far more informative than simply stating whether the result was significant.

Common Mistakes

“p < 0.05 Means the Effect Is Important”

Not necessarily.

Statistical significance and practical importance are different concepts.

“p > 0.05 Means There Is No Effect”

Not necessarily.

The study may have insufficient power or the estimate may be imprecise.

Reporting Only the P-Value

A p-value without the estimated effect provides limited information about magnitude.

Ignoring Confidence Intervals

The confidence interval provides important information about uncertainty.

Confusing Relative and Absolute Effects

A 20% relative improvement may correspond to only a 2-percentage-point absolute improvement.

Treating Cohen’s d Thresholds as Universal

Effect-size benchmarks are contextual guidelines, not laws.

Ignoring Sample Size

Very large samples can make tiny effects statistically significant.

Very small samples can make large effects difficult to detect.

Best Practices

Report Effect Size

Don’t rely exclusively on p-values.

Report Confidence Intervals

Show the uncertainty around your estimate.

Report Absolute Effects

They are often easier to interpret.

Consider Relative Effects

Use them when they provide useful additional context.

Include Statistical Significance

It still provides useful evidence when correctly interpreted.

Define Practical Importance

Before an experiment, when possible, determine what size of effect would actually matter.

Consider Statistical Power

Design studies with enough observations to detect effects that matter.

Use Domain Knowledge

Statistical results need real-world interpretation.

Conclusion

Effect size and statistical significance answer different questions.

Statistical significance helps assess whether the observed data provide sufficient evidence against a specified null hypothesis.

Effect size tells you the magnitude of the observed difference or relationship.

Neither should be interpreted in isolation.

A statistically significant result can have a trivial effect.

A large estimated effect can fail to reach statistical significance when a study is small or highly variable.

A stronger analysis therefore combines:

Effect size + confidence interval + statistical evidence + practical context.

When reporting results, don’t stop at:

“The result was statistically significant.”

Instead, explain:

How large was the effect, how uncertain is the estimate, and does the magnitude actually matter?

That is the difference between simply running a statistical test and meaningfully interpreting its results.

Frequently Asked Questions