Before running an experiment, one of the most important questions is:

How many observations do I need to reliably detect an effect?

This is where statistical power comes in.

Imagine an e-commerce company wants to test whether a redesigned checkout page increases conversions.

The team could run an experiment with 100 visitors, 1,000 visitors, or 100,000 visitors.

But how do they know which sample size is appropriate?

A small experiment may fail to detect a real improvement.

A very large experiment may waste time, money, and resources.

Statistical power helps researchers design experiments that have a reasonable chance of detecting a meaningful effect when that effect actually exists.

Statistical power is the probability that an experiment will correctly reject the null hypothesis when a real effect exists.

It is commonly represented as:

Power = 1 − β

where β is the probability of a Type II error.

Before an experiment, power analysis typically uses four key inputs:

• Significance level (α)
• Statistical power
• Effect size
• Sample size

Usually, if you know three of these quantities, you can calculate the fourth.

A common target is:

α = 0.05
Power = 0.80

meaning the experiment is designed to have an 80% chance of detecting the specified effect.

What Is Statistical Power?

Statistical power measures how capable a statistical test is of detecting a real effect.

Suppose a new website design genuinely increases conversion rates.

A sufficiently powered experiment has a good chance of detecting that improvement.

A poorly powered experiment might conclude:

“There is no statistically significant difference.”

even though the new design actually works.

This is called a Type II error or false negative.

Statistical Power and Type II Errors

Statistical power is directly related to the Type II error probability.

Power = 1 − β

If:

β = 0.20

then:

Power = 0.80

So an experiment with 80% power has a 20% probability of failing to detect the specified effect, assuming the effect truly exists and the assumptions of the statistical test hold.

Type I vs Type II Errors

There are two major error types.

Error	Meaning
Type I error	Detecting an effect that does not exist
Type II error	Failing to detect an effect that does exist

The significance level controls the Type I error rate.

Power is related to the Type II error rate.

α → Type I error
β → Type II error
1 − β → Statistical power

The Four Main Inputs

Power analysis usually involves four quantities.

1. Sample Size

The number of observations included in the experiment.

For example:

n = 1,000

2. Effect Size

The size of the effect you want to detect.

For example:

Conversion rate:
5% → 6%

The absolute improvement is:

1 percentage point

The relative improvement is:

20%

Statistical power calculations need a properly defined effect-size measure appropriate for the statistical test.

3. Significance Level

The significance level, usually represented by α, defines the threshold for statistical significance.

A common choice is:

α = 0.05

This means the test is designed with a 5% Type I error rate under its assumptions.

4. Desired Power

Researchers commonly choose:

80%

or:

90%

Higher power generally requires a larger sample.

Why Effect Size Matters So Much

Suppose you want to detect an enormous improvement:

5% → 15%

That effect is relatively easy to detect.

Now consider:

5% → 5.2%

The difference is much smaller.

You generally need more observations to distinguish a small effect from random variation.

This leads to an important principle:

Smaller effects generally require larger samples to detect reliably.

The Relationship Between Sample Size and Power

Holding other factors constant:

Larger sample
      ↓
Smaller uncertainty
      ↓
Greater ability to detect effects
      ↓
Higher statistical power

But increasing sample size isn’t the only way power changes.

Power also depends on:

• Effect size
• Variability
• Significance level
• Test type
• One-sided vs two-sided testing
• Experimental design

A Simple Example

Suppose an online store currently has a conversion rate of:

5%

The company wants to test a new checkout design.

The smallest improvement worth detecting is:

5% → 6%

The team chooses:

Significance level = 0.05
Power = 0.80

The question becomes:

How large should the experiment be?

This is a sample size calculation based on the desired effect.

The answer depends on the statistical test and assumptions, but the workflow is:

Baseline conversion
        ↓
Minimum detectable effect
        ↓
Significance level
        ↓
Desired power
        ↓
Sample size calculation

What Is a Minimum Detectable Effect?

The minimum detectable effect (MDE) is the smallest effect your experiment is designed to detect at the chosen significance level and power.

For example:

Baseline conversion = 5%
MDE = +1 percentage point

means the experiment is designed to detect a change from roughly:

5% → 6%

under the specified assumptions.

Choosing the MDE is a business decision as much as a statistical one.

If an improvement smaller than 0.1 percentage points would have no meaningful business value, there may be little reason to design an experiment specifically to detect it.

Power Analysis Before an Experiment

The most common type of power analysis performed before an experiment is a priori power analysis.

The workflow is:

Business Question
      ↓
Choose Meaningful Effect
      ↓
Choose α
      ↓
Choose Desired Power
      ↓
Calculate Required Sample
      ↓
Run Experiment

This is different from calculating power after seeing the experimental results.

A Priori Power Analysis

Suppose you’re planning an A/B test.

You choose:

α = 0.05
Power = 0.80
Expected effect = 5% relative improvement

You then calculate how many observations are needed.

This is useful because the sample size is determined before the outcome is known.

Post-Hoc Power Analysis

After an experiment, researchers sometimes calculate “observed power” using the estimated effect and sample size.

However, post-hoc power calculations can be misleading because the observed effect is itself noisy.

For evaluating a completed experiment, confidence intervals and the estimated effect are often more informative than trying to reinterpret the p-value through post-hoc power.

The most useful role for traditional power analysis is generally experiment planning.

Statistical Power for A/B Tests

A/B testing is one of the most common applications.

Suppose:

Control:
10,000 users
Conversion = 5%

Treatment:
10,000 users
Conversion = 5.5%

The question isn’t simply:

“Is the difference statistically significant?”

Before running the experiment, you should ask:

“How many users do we need to have a reasonable chance of detecting a 0.5 percentage-point improvement?”

This is where power analysis helps.

Power for a Two-Proportion Test

For conversion experiments, you’re often comparing two proportions.

Let:

p₁ = control conversion rate
p₂ = treatment conversion rate

The effect is related to:

p₂ − p₁

For example:

p₁ = 0.05
p₂ = 0.06

The difference is:

0.01

or one percentage point.

The required sample size depends on the baseline rate, target difference, α, desired power, and whether the test is one-sided or two-sided.

Approximate Sample Size Formula for Two Proportions

For a two-group comparison of proportions with equal group sizes, a commonly used approximation is:

n ≈
[
z_(1−α/2) √(2p̄(1−p̄))
+
z_(1−β) √(p₁(1−p₁) + p₂(1−p₂))
]²
/
(p₂ − p₁)²

where:

p̄ = (p₁ + p₂) / 2

and:

• α is the significance level
• β is the Type II error probability
• 1 − β is power
• z represents the corresponding standard-normal quantile
• n is the approximate sample size per group

This is an approximation, so software-based power calculations are usually preferable for real experiment planning.

Why the Significance Level Matters

Suppose you use:

α = 0.05

Now consider using:

α = 0.01

The second threshold is more stringent.

All else equal, requiring stronger evidence for significance generally means you need more observations to maintain the same statistical power.

There is therefore a trade-off between:

False-positive control
        ↕
Required sample size
        ↕
Statistical power

One-Sided vs Two-Sided Tests

Power calculations also depend on whether your hypothesis is one-sided or two-sided.

Two-Sided

You want to detect either:

Increase
OR
Decrease

One-Sided

You are specifically testing for an effect in one direction.

For example:

New design improves conversion.

The choice should be made based on the scientific or business question before examining the results.

A one-sided test should not be chosen simply because it makes statistical significance easier to achieve after seeing the data.

Power and Variability

Higher variability generally makes effects harder to detect.

Consider two experiments.

Experiment A

Low variability

Experiment B

High variability

If both have the same effect size and sample size, Experiment A may have greater power.

Reducing measurement noise can therefore improve statistical efficiency without simply collecting more observations.

Power in Continuous Outcomes

Power analysis isn’t limited to conversion rates.

Suppose you’re testing whether a new recommendation algorithm increases average order value.

Control:

Mean = $50

Treatment:

Mean = $53

You need to consider:

• Expected mean difference
• Standard deviation
• Sample size
• Significance level
• Desired power

For a two-group comparison of means, the standardized effect size is often expressed using Cohen’s d:

d = (μ₂ − μ₁) / σ

where the denominator represents an appropriate measure of variability.

Cohen’s d

Cohen’s d expresses a difference relative to variability.

For example:

Mean difference = 5
Standard deviation = 10

gives:

d = 0.5

The same absolute difference can be easy or difficult to detect depending on the amount of variability.

A difference of 5 units means something very different when the standard deviation is 2 versus 100.

Power for Correlation

Power analysis can also be used for correlation studies.

Suppose you want to determine whether:

Advertising spend

is associated with:

Sales

You need to consider the expected correlation magnitude, significance level, desired power, and sample size.

Small expected correlations generally require larger samples.

Power for Regression

For regression experiments, power can involve:

• Number of predictors
• Expected effect size
• Sample size
• Variance explained
• Significance level
• Desired power

Adding many predictors can increase the amount of data required.

A model with 20 predictors generally requires more observations than a simple model with one predictor if you want stable estimates and sufficient power.

Power and Multiple Testing

Suppose you test:

1 hypothesis

and then:

100 hypotheses

The probability of finding a statistically significant result by chance increases when many hypotheses are tested without appropriate correction.

Multiple-testing procedures can control false discoveries, but they can also affect the effective threshold and therefore power.

This is another reason to define primary outcomes before the experiment begins.

Power in Machine Learning Experiments

Statistical power is also relevant to machine learning experiments.

Suppose you’re comparing two recommendation models.

Model A:

CTR = 4.2%

Model B:

CTR = 4.3%

The difference might be real—or it might be random variation.

If your experiment is too small, you may not have enough statistical power to distinguish the two systems.

The same principle applies to:

• Model accuracy
• Conversion rate
• Revenue per user
• Retention
• Latency
• Engagement
• Error rates

Power Does Not Mean Probability the Hypothesis Is True

A common misunderstanding is:

“An 80% powered experiment means there is an 80% probability that my hypothesis is true.”

That’s incorrect.

Power means that under a specified alternative effect and the assumptions of the statistical model, the test has an 80% probability of rejecting the null hypothesis.

It is not the probability that the alternative hypothesis is true.

Power Is Conditional on an Effect Size

This is one of the most important concepts.

There isn’t simply one universal power value for an experiment.

Power depends on the effect you’re trying to detect.

For example:

Effect = 10%
→ High power

Effect = 5%
→ Moderate power

Effect = 1%
→ Low power

assuming the same sample size and other parameters.

This is why power calculations should specify the effect size.

Statistical Power Curve

Instead of calculating power for only one effect size, you can examine how power changes across different effects.

For example:

Effect size
    ↓
Power

You might find:

Effect Size	Power
1%	0.28
2%	0.48
3%	0.67
4%	0.81
5%	0.91

This helps decision-makers understand what their experiment can realistically detect.

How to Calculate Power With Python

Python libraries can calculate statistical power for many common tests.

For example, statsmodels provides power-analysis tools.

For a two-sample comparison of means:

from statsmodels.stats.power import TTestIndPower

analysis = TTestIndPower()

sample_size = analysis.solve_power(
    effect_size=0.5,
    alpha=0.05,
    power=0.80,
    alternative="two-sided"
)

print(sample_size)

This solves for the required sample size given the effect size, significance level, and desired power.

Calculating Power When Sample Size Is Known

Sometimes the sample size is already fixed.

For example:

n = 500 per group

You can instead solve for power.

from statsmodels.stats.power import TTestIndPower

analysis = TTestIndPower()

power = analysis.solve_power(
    effect_size=0.5,
    nobs1=500,
    alpha=0.05,
    ratio=1,
    alternative="two-sided"
)

print(power)

This answers:

“Given this sample size and expected effect, how much statistical power do we have?”

Sample Size Is Not the Only Design Decision

Power analysis should be part of broader experimental design.

Also consider:

• Randomization
• Control group design
• Measurement quality
• Outcome definition
• Expected variance
• Experiment duration
• Seasonality
• Attrition
• Missing data
• Multiple testing
• Practical significance

A perfectly calculated sample size cannot rescue a poorly designed experiment.

Statistical vs Practical Significance

A large experiment can detect extremely small effects.

Suppose:

Conversion:
5.000% → 5.005%

With enough observations, the difference might become statistically significant.

But is it useful?

Perhaps not.

This creates an important distinction:

Statistical significance asks whether the observed effect is unlikely under the null hypothesis.

Practical significance asks whether the effect is large enough to matter in the real world.

Power analysis should therefore be based on a meaningful effect size, not merely the smallest mathematically detectable difference.

What Happens When an Experiment Is Underpowered?

An underpowered experiment may produce:

• Wide confidence intervals
• Unstable estimates
• Non-significant results despite real effects
• Difficulty reproducing findings
• Increased uncertainty

A non-significant result from an underpowered study should not automatically be interpreted as proof that there is no effect.

It may simply mean the experiment did not have enough information to detect the effect.

What Happens When an Experiment Is Overpowered?

An extremely large experiment can detect tiny effects that have little practical importance.

It can also consume:

• More users
• More money
• More engineering resources
• More time

The goal isn’t maximum power at any cost.

The goal is an appropriate design that can detect effects that matter.

A Practical Pre-Experiment Checklist

Before launching an experiment, define:

1. Primary Metric

What exactly are you measuring?

Conversion rate
Revenue per user
Retention
Average order value

2. Baseline

What is the current performance?

Conversion = 5%

3. Minimum Meaningful Effect

What improvement would justify the experiment?

+1 percentage point

4. Significance Level

For example:

α = 0.05

5. Desired Power

For example:

80%

6. Test Type

For example:

Two-proportion test

7. Required Sample

Use an appropriate power-analysis method.

8. Experiment Duration

Make sure the required sample can realistically be collected.

Common Mistakes

Choosing Sample Size Arbitrarily

“Let’s just use 1,000 users” isn’t a statistical design.

The required sample should be connected to the expected effect and desired power.

Using the Largest Possible Sample

More data isn’t automatically better if the additional observations provide little practical value.

Ignoring Baseline Rates

For proportion-based experiments, the baseline rate affects sample-size requirements.

Using an Unrealistically Tiny Effect

Designing an experiment to detect an effect so small that it has no business value can create unnecessary sample requirements.

Changing the MDE After Seeing Results

The minimum detectable effect should be established before examining the outcome.

Stopping Because Results Look Significant

Repeatedly checking results and stopping when a p-value crosses a threshold can inflate false-positive rates unless the experiment uses an appropriate sequential design.

Ignoring Attrition

If participants may drop out or data may be unusable, recruit enough participants to account for expected losses.

Best Practices

Define the MDE Before the Experiment

The MDE should represent a meaningful effect, not simply a convenient number.

Choose Power Before Collecting Data

Common targets include 80% or 90%, depending on the consequences of false negatives.

Use Appropriate Statistical Tests

A power calculation for a t-test isn’t automatically appropriate for a conversion experiment.

Account for Multiple Comparisons

If many hypotheses are tested, incorporate the testing strategy into the design.

Consider Real-World Constraints

Your theoretical sample size must be achievable within your experiment’s timeline and budget.

Report Assumptions

Document:

Baseline
MDE
α
Power
Test type
Sample size

This makes the experiment reproducible.

Conclusion

Statistical power is one of the most useful concepts for planning experiments.

It answers a practical question:

If a meaningful effect really exists, how likely is my experiment to detect it?

Before collecting data, you can use power analysis to connect:

Effect size + significance level + desired power → required sample size

The most important step isn’t simply choosing an 80% or 90% power target.

It’s defining an effect size that actually matters.

A well-designed experiment should have enough observations to detect meaningful changes without collecting vastly more data than necessary.

In practice, power analysis turns experiment design from:

“How much data should we collect?”

into:

“How much data do we need to reliably detect an effect that matters?”

That is a much better question.

Frequently Asked Questions