Permutation Tests Explained With Python

When analysing data, one of the most important questions you may need to answer is whether an observed difference is meaningful or simply the result of random variation.

For example, imagine an online business introduces a new website design and notices that its conversion rate increases. Does the new design genuinely improve conversions, or could the increase have happened by chance?

Statistical hypothesis testing helps analysts investigate questions like this. However, many traditional statistical tests rely on assumptions about the distribution of data. When those assumptions are not satisfied, choosing the right test can become difficult.

This is where permutation tests become useful.

A permutation test is a statistical technique that evaluates whether an observed result is unusual by repeatedly rearranging the data and comparing the original result with results generated under the assumption that there is no real effect.

In this guide, you will learn what permutation tests are, how they work, when to use them, and how to implement them in Python using practical examples.

What Is a Permutation Test?

A permutation test is a non-parametric statistical hypothesis test that determines whether an observed difference, relationship, or effect could reasonably occur by chance.

Instead of assuming that your data follows a particular probability distribution, a permutation test creates a reference distribution by rearranging the observed data.

The fundamental idea is simple:

If two groups are genuinely similar under the null hypothesis, their observations should be interchangeable. Randomly rearranging the group labels should therefore produce results similar to the original result.

If the observed difference is much larger or smaller than most differences produced by these rearrangements, there may be evidence of a statistically significant effect.

A simple example

Suppose a company wants to compare the performance of two marketing campaigns.

• Campaign A generates an average of 120 website visits per day.
• Campaign B generates an average of 150 website visits per day.

The observed difference is:

150 − 120 = 30 visits

The company wants to know whether this difference represents a genuine improvement or whether random variation could explain it.

A permutation test would:

1. Combine the observations from both campaigns.
2. Randomly redistribute the observations into two groups.
3. Calculate the difference between the groups.
4. Repeat the process thousands of times.
5. Compare the original difference with the simulated differences.

This produces a distribution of differences expected under the null hypothesis.

If a difference of 30 is rarely produced by random rearrangements, the result provides evidence against the null hypothesis.

Understanding the Hypotheses

Like other statistical tests, permutation tests begin with a null hypothesis and an alternative hypothesis.

Null hypothesis (H₀)

The null hypothesis states that there is no meaningful difference or effect between the groups being compared.

For the marketing campaign example:

H₀: Campaign A and Campaign B have the same underlying performance.

Alternative hypothesis (H₁)

The alternative hypothesis states that a difference or effect exists.

There are three common ways to define the alternative hypothesis.

Test type	Alternative hypothesis
Two-sided	The groups have different outcomes.
Greater-than	Campaign B performs better than Campaign A.
Less-than	Campaign B performs worse than Campaign A.

Your choice depends on the research question.

A two-sided test is appropriate when you want to detect a difference in either direction. A one-sided test is appropriate when you have a justified directional hypothesis established before examining the results.

How Does a Permutation Test Work?

A permutation test follows a repeatable process.

Step 1: Calculate the observed statistic

First, choose a statistic that represents the effect you want to investigate.

Common examples include:

• Difference between two means.
• Difference between two medians.
• Difference between proportions.
• Correlation coefficient.
• Difference in model performance.
• Difference in conversion rates.

For two independent groups, the difference in means is a common choice.

[
T_{\text{observed}} = \bar X_B – \bar X_A
]

Step 2: Combine the observations

Under the null hypothesis, the group labels are interchangeable for an appropriate randomisation design.

The observations are combined into one dataset while preserving the original group sizes.

Step 3: Randomly shuffle the group labels

The combined observations are randomly rearranged and divided into two groups.

This creates a simulated dataset in which the original group assignment has been removed.

Step 4: Calculate the statistic again

For every rearrangement, calculate the same statistic used in the original analysis.

For example, calculate the difference between the two simulated group means.

Step 5: Repeat the process

Repeat the randomisation many times to create a permutation distribution.

The more permutations you perform, the more precise your Monte Carlo approximation of the permutation p-value becomes.

Step 6: Calculate the p-value

The p-value measures how frequently the simulated statistics are at least as extreme as the observed statistic under the null hypothesis.

For a two-sided test:

[
p = \frac{1 + \sum_{i=1}^{B} I(|T_i| \geq |T_{\text{observed}}|)}{B+1}
]

Where:

• (B) is the number of permutations.
• (T_i) is the statistic from permutation (i).
• (I) equals 1 when the condition is true and 0 otherwise.
• (T_{\text{observed}}) is the original statistic.

The addition of one to the numerator and denominator is a common correction for Monte Carlo permutation tests. It prevents reporting a p-value of zero simply because none of the sampled permutations was as extreme as the observed result.

Permutation Test vs Traditional Statistical Tests

Permutation tests are often discussed alongside parametric tests such as the independent t-test.

Although they can answer similar research questions, their approaches are different.

Feature	Permutation test	Independent t-test
Main approach	Randomly rearranges observations or labels	Uses a theoretical t-distribution
Distribution assumptions	Does not require normally distributed observations for a valid randomisation test	Classical version assumes normal populations; it is often reasonably robust under suitable conditions
Sample size	Can be computationally expensive for large permutation counts	Usually computationally inexpensive
Flexibility	Can test many statistics	Designed primarily for comparing means
Interpretation	Compares observed statistics with a permutation distribution	Compares a test statistic with a theoretical distribution

A permutation test is not automatically assumption-free.

Its validity depends on the randomisation or exchangeability assumptions behind the test. For example, simply shuffling observations between groups is not appropriate for every experimental design.

Implementing a Permutation Test in Python

Python provides several ways to perform permutation tests. You can write the randomisation procedure yourself or use statistical libraries.

We will start with a practical example using NumPy and Matplotlib.

Example: Comparing Two Marketing Campaigns

Suppose an organisation runs two marketing campaigns and records the daily number of conversions generated by each campaign.

The goal is to determine whether the observed difference in average conversions is statistically significant.

Step 1: Import the required libraries

import numpy as np
import matplotlib.pyplot as plt

NumPy will handle the calculations and random sampling, while Matplotlib will help us visualise the permutation distribution.

Step 2: Create the sample data

campaign_a = np.array([
    18, 20, 22, 19, 17, 21, 23, 20, 18, 19
])

campaign_b = np.array([
    24, 26, 23, 28, 25, 27, 22, 29, 26, 25
])

These values represent daily conversion counts for the two campaigns.

Step 3: Calculate the observed difference

observed_difference = (
    np.mean(campaign_b) - np.mean(campaign_a)
)

print("Observed Difference:", observed_difference)

The observed difference is the statistic we will compare against the results of random permutations.

Step 4: Create the permutation test function

def permutation_test(group_a, group_b, n_permutations=10000):
    rng = np.random.default_rng(42)

    observed = np.mean(group_b) - np.mean(group_a)

    combined = np.concatenate([group_a, group_b])

    n_a = len(group_a)
    permutation_differences = []

    for _ in range(n_permutations):
        shuffled = rng.permutation(combined)

        perm_a = shuffled[:n_a]
        perm_b = shuffled[n_a:]

        difference = np.mean(perm_b) - np.mean(perm_a)

        permutation_differences.append(difference)

    permutation_differences = np.array(
        permutation_differences
    )

    p_value = (
        1 + np.sum(
            np.abs(permutation_differences) >= abs(observed)
        )
    ) / (n_permutations + 1)

    return observed, p_value, permutation_differences

Let’s understand what happens inside this function.

• default_rng(42) creates a reproducible random number generator.
• np.concatenate() combines both campaign datasets.
• rng.permutation() randomly rearranges the observations.
• The shuffled observations are divided into two groups with their original sample sizes.
• The difference between the group means is calculated.
• The process is repeated 10,000 times.
• The p-value is calculated by comparing the simulated differences with the observed difference.

The function returns three results: the observed difference, the p-value, and the complete permutation distribution.

Step 5: Run the test

observed, p_value, distribution = permutation_test(
    campaign_a,
    campaign_b,
    n_permutations=10000
)

print("Observed Difference:", observed)
print("Permutation p-value:", p_value)

The result will show the observed difference and its estimated two-sided p-value.

Because the campaign observations are deliberately separated in this example, the permutation distribution should provide strong evidence against the null hypothesis.

Step 6: Visualise the permutation distribution

A visualisation makes it easier to understand how unusual the observed result is.

plt.figure(figsize=(10, 6))

plt.hist(
    distribution,
    bins=40,
    edgecolor="black"
)

plt.axvline(
    observed,
    linestyle="--",
    linewidth=2,
    label="Observed Difference"
)

plt.axvline(
    -observed,
    linestyle="--",
    linewidth=2,
    label="Negative Observed Difference"
)

plt.title("Permutation Test Distribution")
plt.xlabel("Difference in Mean Conversions")
plt.ylabel("Frequency")
plt.legend()

plt.show()

The histogram represents the differences generated by random permutations.

The vertical lines represent the observed difference and its negative counterpart. For a two-sided test, we are interested in simulated results at least as extreme in absolute value as the observed statistic.

If very few simulated results reach these values, the observed difference is unusual under the null hypothesis.

Using SciPy for Permutation Tests

Writing your own function is useful for understanding how permutation tests work. However, you do not always need to implement the procedure manually.

The SciPy library includes a permutation test function that supports different experimental designs.

You can use it with the same campaign data.

from scipy.stats import permutation_test
import numpy as np

def mean_difference(a, b):
    return np.mean(b) - np.mean(a)

result = permutation_test(
    data=(campaign_a, campaign_b),
    statistic=mean_difference,
    permutation_type="independent",
    alternative="two-sided",
    n_resamples=10000,
    random_state=42
)

print("Statistic:", result.statistic)
print("P-value:", result.pvalue)

The permutation_type="independent" argument indicates that the observations belong to two independent samples.

Other important arguments include:

• alternative: Specifies whether the test is two-sided, greater-than, or less-than.
• n_resamples: Controls the number of random permutations.
• random_state: Helps make the randomisation reproducible.

For paired observations, you can use an appropriate paired permutation procedure rather than treating the two samples as independent.

SciPy also returns a confidence interval for the test statistic’s permutation distribution. This should not automatically be interpreted as a confidence interval for the population effect; use a suitable effect-estimation method when that is your objective.

When Should You Use a Permutation Test?

Permutation tests are useful in several areas of data analysis and research.

A/B testing

Businesses frequently use A/B testing to compare two versions of a website, advertisement, landing page, or product feature.

A permutation test can help determine whether the observed difference in a chosen outcome is consistent with random assignment under the null hypothesis.

For example, an analyst may compare the average revenue per visitor between two versions of an online store.

Comparing machine learning models

Suppose you develop two classification models and want to compare their performance.

You could investigate differences in accuracy, F1-score, or another performance metric.

However, predictions on the same test dataset are not independent observations. The permutation procedure must preserve the evaluation structure and account for paired predictions. Simply shuffling individual model scores or treating repeated predictions as independent can produce misleading results.

Data science research

Researchers may use permutation tests to investigate differences between experimental groups, correlations, or other statistical relationships.

They are particularly useful when the assumptions of a conventional parametric test are questionable but a valid randomisation scheme is available.

Comparing customer behaviour

A business may want to know whether customers from two regions spend significantly different amounts.

A permutation test can compare an appropriate statistic, such as the difference in mean spending, provided the sampling and exchangeability assumptions are satisfied.

Different Types of Permutation Tests

The appropriate permutation procedure depends on the structure of your data and research question.

Independent permutation test

This approach is used when comparing two independent groups.

For example, customers assigned to two different marketing campaigns.

The observations are pooled and reassigned while preserving the original group sizes.

Paired permutation test

A paired test is suitable when observations are naturally matched.

For example, measuring employee productivity before and after training.

Instead of randomly assigning individual measurements to unrelated groups, the test works with the paired differences and applies a valid random sign-flipping procedure under the relevant null hypothesis.

Correlation permutation test

A permutation test can also investigate whether two variables have a meaningful association.

For example, you may want to examine whether advertising expenditure is associated with sales revenue.

One possible approach is to keep one variable fixed and randomly rearrange the values of the other, then recalculate the correlation.

This approach requires an appropriate exchangeability assumption. Time series, clustered observations, and other structured datasets may require restricted permutations.

Common Mistakes to Avoid

Although permutation tests are relatively straightforward, incorrect implementation can lead to misleading conclusions.

1. Ignoring the experimental design

Not every dataset can be freely shuffled.

For example, measurements collected from the same individuals before and after an intervention are paired. Treating them as independent observations ignores the relationship between measurements.

Always identify whether your observations are independent, paired, clustered, or otherwise structured.

2. Using too few permutations

Running only 100 permutations may provide a rough indication of the result, but it gives a relatively coarse p-value estimate.

For many practical applications, 10,000 or more permutations provide a useful starting point. The appropriate number depends on the desired precision, computational resources, and significance threshold.

3. Misinterpreting the p-value

A p-value of 0.03 does not mean there is a 97% probability that your hypothesis is correct.

It means that, under the null hypothesis and the assumptions of the test, a result at least as extreme as the observed result would occur with an estimated probability of 3%.

4. Confusing statistical significance with practical importance

A statistically significant result does not necessarily represent a meaningful business improvement.

For example, an increase of 0.01% in conversion rate could be statistically significant in a very large dataset but have little practical value.

Always report the effect size alongside the p-value.

5. Ignoring multiple comparisons

If you perform dozens of permutation tests, some may produce small p-values by chance.

When conducting multiple hypothesis tests, consider appropriate corrections, such as the Holm procedure or false discovery rate control.

Best Practices for Reliable Permutation Testing

To improve the reliability of your analysis:

1. Define your null and alternative hypotheses before running the test.
2. Select a test statistic that directly answers your research question.
3. Understand the experimental design and identify valid exchangeability assumptions.
4. Use an appropriate permutation strategy for independent, paired, or structured observations.
5. Perform enough permutations to obtain a sufficiently precise p-value.
6. Set a random seed when reproducibility is important.
7. Report the observed statistic, p-value, sample sizes, and effect size.
8. Visualise the permutation distribution when it helps explain the result.
9. Account for multiple comparisons when conducting several tests.
10. Interpret statistical significance in the context of the actual business or research problem.

A well-designed permutation test can provide useful statistical evidence without requiring you to rely on a particular theoretical distribution. However, the quality of the conclusion still depends on the quality of the data and the validity of the randomisation procedure.

Conclusion

Permutation tests provide a flexible way to investigate statistical significance by comparing observed results with results generated through random rearrangements of data.

They are particularly useful when analysing differences between groups, evaluating experiments, investigating relationships, and testing model performance.

With Python libraries such as NumPy and SciPy, you can implement permutation tests without building complex statistical procedures from scratch.

The most important lesson is that permutation tests are not simply about shuffling data thousands of times. You must understand the null hypothesis, preserve the structure of your experiment, choose an appropriate statistic, and interpret the results correctly.

When applied properly, permutation tests can become a valuable addition to your statistical analysis toolkit.

Frequently Asked Questions