Machine Learning Data Leakage From Feature Engineering: Common Mistakes and How to Prevent Them

Feature engineering is one of the most important steps in building a machine learning model. It transforms raw data into useful variables that help algorithms identify patterns, make predictions, and solve business problems.

However, feature engineering can also introduce a subtle problem: data leakage.

A model may achieve excellent accuracy during training and validation but perform poorly when deployed because its features contain information that would not have been available when the prediction was supposed to be made.

This problem is particularly dangerous because the resulting model can appear successful throughout the development process. The issue may only become visible when real-world predictions fail.

For example, imagine building a model to predict whether a customer will cancel a subscription next month. If one of the engineered features includes the date on which the customer actually cancelled, the model may learn to recognize cancellations rather than predict them.

Other leakage problems are less obvious. Calculating average customer spending using the entire dataset before splitting it, applying target encoding incorrectly, or creating rolling features that include future observations can all produce misleading evaluation results.

In this guide, you will learn how machine learning data leakage occurs during feature engineering, how to detect it using Python, and how to build a workflow that produces more reliable models.

What Is Data Leakage in Machine Learning?

Data leakage occurs when information unavailable at the intended prediction time influences model training or evaluation.

As a result, the model can learn patterns that would not be accessible when making genuine predictions.

Consider a loan default prediction model. The goal is to estimate whether an applicant will default based on information available when the loan is approved.

If a feature contains the number of missed payments recorded six months after approval, that feature leaks future information into the model.

The model may achieve an impressive evaluation score, but the feature cannot legitimately be used for the original prediction task.

Data leakage can occur in several places:

• Feature engineering
• Train-test splitting
• Data preprocessing
• Target encoding
• Time-series feature creation
• Dataset joins
• Model evaluation
• Data collection and label generation

The central question is not simply whether a feature is correlated with the target. It is whether the information used to construct that feature would genuinely have been available at prediction time.

Why Feature Engineering Can Introduce Data Leakage

Feature engineering often involves aggregations, transformations, statistical summaries, categorical encodings, and combinations of existing variables.

These operations can unintentionally introduce information from the target variable, the future, or the validation dataset.

Consider a model that predicts customer churn. A data scientist might create features such as:

• Average monthly spending
• Number of support tickets
• Days since the last purchase
• Customer lifetime value
• Number of complaints
• Recent product activity

These features may be useful, but their validity depends on how and when they are calculated.

If average spending is calculated using transactions recorded after the prediction date, it may contain future information. If customer lifetime value includes purchases made after cancellation, it may no longer represent what was known when the prediction was made.

Feature engineering must therefore respect the timeline and information boundaries of the prediction problem.

1. Leakage From Calculating Features Before the Train-Test Split

One common mistake is calculating data-dependent transformations using the entire dataset before separating training and test data.

Suppose you want to standardize a numerical feature using its mean and standard deviation.

A developer might write:

import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

X = df.drop(columns=["target"])
y = df["target"]

scaler = StandardScaler()

X_scaled = scaler.fit_transform(X)

X_train, X_test, y_train, y_test = train_test_split(
    X_scaled,
    y,
    test_size=0.2,
    random_state=42
)

The problem is that StandardScaler learns the mean and standard deviation from the entire dataset, including the eventual test set.

The target values are not directly used, so this is not necessarily severe target leakage. However, the preprocessing has allowed information about the test feature distribution to influence training.

In a strict evaluation workflow, the test set should remain isolated until evaluation.

The correct approach

Split the data first, then fit the scaler on the training data only.

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

X = df.drop(columns=["target"])
y = df["target"]

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42
)

scaler = StandardScaler()

X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

The distinction is important:

• fit_transform() learns preprocessing parameters and applies them.
• transform() applies parameters learned from the training data.

The test set must not influence the parameters used to prepare the training data.

Scaling may be unnecessary for some algorithms, including many tree-based models, but the same principle applies to other learned preprocessing steps.

2. Leakage Through Feature Selection

Feature selection identifies which variables should be used by a model.

Some feature-selection methods rely on statistical relationships between predictors and the target. If you perform feature selection using the complete dataset before splitting, information from the test labels can influence which features are selected.

For example:

from sklearn.feature_selection import SelectKBest, f_classif

selector = SelectKBest(
    score_func=f_classif,
    k=10
)

X_selected = selector.fit_transform(X, y)

Because the selector uses y, it learns which features appear most predictive across the entire dataset.

If the data is split afterward, the test labels have already influenced feature selection.

This can make the final evaluation overly optimistic, particularly when the dataset contains many features and relatively few observations.

Preventing feature-selection leakage

Split the dataset before feature selection and fit the selector only on the training data.

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42,
    stratify=y
)

selector = SelectKBest(
    score_func=f_classif,
    k=10
)

X_train_selected = selector.fit_transform(
    X_train,
    y_train
)

X_test_selected = selector.transform(X_test)

For model selection and hyperparameter tuning, feature selection should also occur within each cross-validation training fold. A scikit-learn pipeline is generally the safest way to achieve this.

3. Leakage From Target Encoding

Target encoding replaces a categorical value with a statistic calculated from the target variable.

For example, suppose you want to predict whether a customer will default. You could encode each city using the average default rate among customers in that city.

The problem is that an encoding calculated using every observation can include the target value of the same row being encoded.

For a rare category, the encoding may become almost identical to the target itself.

Consider this example:

import pandas as pd

data = pd.DataFrame({
    "city": ["A", "A", "B", "C", "C"],
    "default": [0, 1, 0, 1, 1]
})

data["city_default_rate"] = (
    data.groupby("city")["default"]
    .transform("mean")
)

print(data)

For city B, there is only one observation. Its encoded default rate becomes exactly 0 because its own target value is the only value used to calculate the statistic.

This does not automatically mean every target-encoding method is invalid. The problem is using information in a way that would not be available for unseen observations and allowing a row’s own label to determine its representation.

Safer target encoding

Several approaches can reduce leakage:

• Fit the encoder using training data only.
• Use out-of-fold target encoding for training observations.
• Apply smoothing to reduce unstable estimates for rare categories.
• Ensure validation and test observations are encoded without using their target values.
• Respect group or time boundaries when ordinary random cross-validation is inappropriate.

For example, scikit-learn provides TargetEncoder in supported versions. Its fit_transform() uses internal cross-fitting for training data, while transform() uses statistics learned during fitting.

from sklearn.preprocessing import TargetEncoder

encoder = TargetEncoder(
    target_type="binary",
    smooth="auto",
    cv=5,
    random_state=42
)

X_train_encoded = encoder.fit_transform(
    X_train[["city"]],
    y_train
)

X_test_encoded = encoder.transform(
    X_test[["city"]]
)

This example assumes a binary target and that city is a categorical column in the training and test datasets.

For grouped or time-dependent data, the default cross-fitting strategy may not respect the required boundaries. In those cases, use a strategy designed for the data’s structure.

4. Leakage From Aggregated Features

Aggregated features are widely used in customer analytics, fraud detection, recommendation systems, and financial modeling.

Examples include:

• Average transaction value per customer
• Total purchases per account
• Average product rating
• Number of previous complaints
• Historical default rate by region
• Total spending during the previous 30 days

These features can be useful, but their construction must respect the prediction timestamp.

Imagine predicting whether a customer will make a purchase tomorrow. If total customer spending is calculated using all available transactions, it may include purchases that occur after the prediction time.

A feature that appears reasonable in a static table may therefore be invalid for a real-time prediction.

A safer approach

Calculate aggregates using only observations available before the prediction timestamp.

For example, suppose a transaction dataset contains customer_id, transaction_date, and amount.

transactions = transactions.sort_values(
    ["customer_id", "transaction_date"]
)

transactions["previous_transaction_total"] = (
    transactions.groupby("customer_id")["amount"]
    .transform(
        lambda values: values.shift(1).cumsum()
    )
)

The shift(1) operation excludes the current transaction from the cumulative total. This can be useful when constructing historical features for a transaction-level prediction.

However, this example assumes the records are ordered correctly and that the prediction is made before the current transaction’s amount is known. If multiple transactions share a timestamp, you must define their ordering and availability carefully.

For production systems, it is often safer to construct features using explicit event-time cutoffs and historical windows rather than relying on an undifferentiated full-dataset aggregate.

5. Leakage From Time-Series Feature Engineering

Time-series models are particularly vulnerable to leakage because the order in which observations become available matters.

Suppose a business wants to forecast next week’s demand using historical sales.

Useful features might include:

• Sales from the previous day
• Sales from the previous seven days
• Rolling average sales
• Previous month’s revenue
• Day of the week
• Holiday indicators known in advance

A mistake occurs when the feature calculation includes the value being predicted or observations that occur afterward.

An incorrect rolling feature

df["rolling_mean"] = (
    df["sales"]
    .rolling(window=7)
    .mean()
)

This calculation includes the current observation in its seven-row window. If the goal is to predict the current sales value before it occurs, that creates leakage.

A safer rolling feature

df["rolling_mean_7"] = (
    df["sales"]
    .shift(1)
    .rolling(window=7)
    .mean()
)

The shift moves the sales history back by one observation, so the feature uses previous values rather than the current target.

This assumes rows are sorted chronologically and each row represents the intended time interval. If intervals are irregular, use a time-based window with an explicit cutoff.

Use chronological evaluation

For forecasting tasks, a random train-test split often fails to represent how the model will be used.

Instead, train on earlier observations and test on later observations.

df = df.sort_values("date")

split_index = int(len(df) * 0.8)

train = df.iloc[:split_index]
test = df.iloc[split_index:]

For robust evaluation, use rolling-origin or expanding-window validation where appropriate. These methods repeatedly train on past data and evaluate on future periods.

Also ensure that every feature is available at the forecast origin. A holiday calendar may be known in advance, while the final number of transactions on that holiday is not.

6. Leakage Through Dataset Joins

Joining datasets can introduce leakage when the joined information was recorded after the prediction time or contains information derived from the target.

Imagine a company wants to predict whether a loan applicant will default at approval time.

The main dataset contains application details, while another dataset contains repayment history.

Joining repayment records to applications may be valid if the records describe the applicant’s earlier loans and those records were available at approval time.

However, joining information about the outcome of the current loan would leak future information.

Before joining datasets, ask:

1. When was each record created?
2. When did the information become available?
3. Does the join introduce future information?
4. Can one row match multiple records and duplicate observations?
5. Is the joined field derived from the outcome being predicted?

Correct joins require both appropriate keys and appropriate temporal boundaries.

For time-dependent applications, an ordinary join on customer ID may be insufficient. The join may also need to enforce that the source record’s effective or availability timestamp is earlier than the prediction timestamp.

7. Leakage From Imputation and Missing-Value Handling

Missing-value imputation replaces missing observations with estimated values.

For example, a numerical column might be filled with its mean.

df["income"] = df["income"].fillna(
    df["income"].mean()
)

If this operation is performed before splitting the dataset, the mean is influenced by the test set.

This is another form of preprocessing leakage because the test feature distribution affects the training transformation.

Use a pipeline for imputation

from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

pipeline = Pipeline([
    (
        "imputer",
        SimpleImputer(strategy="median")
    ),
    (
        "scaler",
        StandardScaler()
    ),
    (
        "model",
        LogisticRegression(max_iter=1000)
    )
])

The pipeline learns the imputation values and scaling parameters during fitting. When you fit it on the training data, the test data does not influence those parameters.

You can then evaluate the complete workflow:

pipeline.fit(X_train, y_train)

predictions = pipeline.predict(X_test)

This pattern is useful because the same learned transformations are applied consistently during training and inference.

8. Detecting Feature Engineering Leakage With Python

Leakage detection requires more than checking model accuracy. The strongest approach combines timeline analysis, preprocessing audits, validation design, and investigation of suspicious features.

Check for features that directly reveal the target

Start by inspecting the dataset’s columns and data types.

print(df.columns.tolist())
print(df.dtypes)

Look for variables that might encode the outcome directly, such as:

• Cancellation date in a churn prediction dataset
• Final repayment status in a default prediction dataset
• Post-treatment measurements in a treatment-response model
• Resolution status in a support-ticket prediction task
• Future sales totals in a demand forecasting model

A feature’s name alone cannot establish leakage. You must examine how it was created and whether it would have been available at prediction time.

Compare training and test performance

A very large gap between training and test performance can indicate overfitting, distribution differences, or leakage-related problems elsewhere in the evaluation process.

from sklearn.metrics import accuracy_score

train_predictions = pipeline.predict(X_train)
test_predictions = pipeline.predict(X_test)

train_accuracy = accuracy_score(
    y_train,
    train_predictions
)

test_accuracy = accuracy_score(
    y_test,
    test_predictions
)

print("Train accuracy:", train_accuracy)
print("Test accuracy:", test_accuracy)

High training accuracy with substantially lower test accuracy can indicate overfitting, but it does not prove leakage.

Conversely, suspiciously high performance on both training and test data may indicate that the test set is not truly independent, a feature reveals the target, or information crossed the evaluation boundary.

For imbalanced classification problems, accuracy may also be misleading. Evaluate metrics such as precision, recall, F1 score, ROC-AUC, or precision-recall AUC according to the use case.

Run a permutation test on the labels

A useful diagnostic is to shuffle the training labels and check whether the complete modeling workflow still produces meaningful predictive performance.

If a model achieves unusually strong performance when labels are randomized, investigate the workflow for leakage, duplicate observations, target-derived features, or an invalid evaluation procedure.

import numpy as np
from sklearn.base import clone
from sklearn.metrics import roc_auc_score

rng = np.random.default_rng(42)

shuffled_y_train = rng.permutation(
    np.asarray(y_train)
)

test_model = clone(pipeline)

test_model.fit(
    X_train,
    shuffled_y_train
)

shuffled_probabilities = (
    test_model.predict_proba(X_test)[:, 1]
)

print(
    "Shuffled-label ROC-AUC:",
    roc_auc_score(
        y_test,
        shuffled_probabilities
    )
)

This example assumes a binary classification pipeline with predict_proba() support.

With properly separated training and test data, shuffled training labels should generally not yield meaningful out-of-sample predictive power. However, one randomized run is not definitive; repeated permutations can provide a more informative reference distribution.

This diagnostic is especially useful when paired with checks for duplicated records and preprocessing steps performed outside the training pipeline.

9. Prevent Leakage With Scikit-Learn Pipelines

A scikit-learn pipeline combines preprocessing and model fitting into one workflow.

Instead of manually scaling, imputing, encoding, and selecting features before splitting the data, you can define these operations once and let the pipeline fit each transformation using only the data supplied to fit().

For example:

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.linear_model import LogisticRegression

numeric_features = ["age", "income"]
categorical_features = ["region", "customer_type"]

numeric_pipeline = Pipeline([
    (
        "imputer",
        SimpleImputer(strategy="median")
    ),
    (
        "scaler",
        StandardScaler()
    )
])

categorical_pipeline = Pipeline([
    (
        "imputer",
        SimpleImputer(strategy="most_frequent")
    ),
    (
        "encoder",
        OneHotEncoder(handle_unknown="ignore")
    )
])

preprocessor = ColumnTransformer([
    (
        "numeric",
        numeric_pipeline,
        numeric_features
    ),
    (
        "categorical",
        categorical_pipeline,
        categorical_features
    )
])

model = Pipeline([
    ("preprocessor", preprocessor),
    (
        "classifier",
        LogisticRegression(max_iter=1000)
    )
])

Fit the model only after creating the train-test split:

model.fit(X_train, y_train)

predictions = model.predict(X_test)

When used with scikit-learn cross-validation and model-selection tools, the pipeline ensures learned preprocessing steps are refitted separately within each training fold.

However, a pipeline cannot automatically fix every leakage problem. A feature already contaminated with future information remains contaminated even if it enters a pipeline. Feature creation itself must respect the prediction timeline.

10. Data Leakage vs. Overfitting vs. Data Drift

These problems can produce similar symptoms but have different causes.

Problem	What happens	Example
Data leakage	Unavailable information influences training or evaluation	A churn feature includes the actual cancellation date
Overfitting	The model learns patterns that do not generalize well	A complex model memorizes noise in training data
Data drift	The distribution of input data changes over time	Customer spending patterns change after a pricing update
Concept drift	The relationship between inputs and the target changes	Customers begin responding differently to the same pricing signals

A model with poor test performance may be overfitting, facing distribution changes, or suffering from another modeling problem.

A model with implausibly high validation performance may have leakage, but other causes are possible.

Correct diagnosis requires reviewing how the data was collected, transformed, split, and evaluated.

11. A Practical Checklist for Leakage-Free Feature Engineering

Before deploying a machine learning model, review the following:

• Define the exact time at which each prediction is made.
• Confirm that every feature would be available at that time.
• Split the data before fitting data-dependent transformations.
• Fit imputers, scalers, encoders, and feature selectors only on training data.
• Use pipelines during cross-validation and model selection.
• Use chronological splits for forecasting and time-dependent prediction.
• Use group-aware splits when observations from the same entity could cross evaluation boundaries.
• Audit joins, aggregations, and target-derived features.
• Check for duplicate or near-duplicate records across training and test sets.
• Compare performance against simple baselines and investigate implausibly high scores.
• Reproduce the feature-generation process as it would run in production.
• Document feature definitions, data availability, and transformation logic.

These checks are especially important when the model will be used for decisions involving money, customers, operations, or other consequential outcomes.

Conclusion

Machine learning data leakage from feature engineering occurs when engineered features contain information that would not be available at the time of prediction or when evaluation data improperly influences model development.

Common sources include preprocessing before splitting, target encoding, future information in time-series features, inappropriate aggregations, dataset joins, and feature selection performed using test labels.

Preventing leakage requires more than improving model code. It requires understanding the prediction timeline, separating training and evaluation information, and reproducing the complete feature-generation process as it would operate in production.

Scikit-learn pipelines help keep preprocessing consistent and prevent many common mistakes. Time-aware feature engineering, appropriate validation strategies, and careful audits of data joins and aggregations address additional risks.

The goal is not simply to achieve a high validation score. It is to build a model whose evaluation reflects the information and conditions it will encounter when making real predictions.

Frequently Asked Questions