A classification model can be accurate and still produce unreliable probabilities.

Suppose a machine learning model predicts that a customer has a 70% probability of churning.

What does that 70% actually mean?

If the model is well calibrated, among many customers receiving predictions close to 70%, approximately 70% should actually churn.

This distinction is important.

A model can correctly rank high-risk customers while producing probabilities that are too high or too low.

This is where model calibration becomes useful.

Calibration adjusts a model’s predicted probabilities so that they better reflect observed outcomes.

In Python, this can be done with tools available in scikit-learn, including CalibratedClassifierCV, calibration curves, sigmoid calibration, and isotonic regression.

To calibrate a classification model in Python:

1. Train your classifier.
2. Generate probability predictions.
3. Measure calibration.
4. Apply a calibration method.
5. Evaluate the calibrated probabilities.
6. Use the calibrated model for probability-based decisions.

A common implementation is:

from sklearn.calibration import CalibratedClassifierCV

calibrated_model = CalibratedClassifierCV(
    model,
    method="sigmoid",
    cv=5
)

calibrated_model.fit(X_train, y_train)

probabilities = calibrated_model.predict_proba(X_test)[:, 1]

The two commonly used calibration methods in scikit-learn are:

• Sigmoid calibration
• Isotonic calibration

What Is Model Calibration?

A classifier is calibrated when its predicted probabilities correspond closely to the actual frequency of outcomes.

For example, imagine a model produces predictions for 1,000 cases.

Among predictions around 80% probability:

Predicted probability: 80%
Actual positive rate: 79%

That is good calibration.

But if:

Predicted probability: 80%
Actual positive rate: 55%

the model is overconfident.

Similarly:

Predicted probability: 40%
Actual positive rate: 65%

would indicate underconfidence in that probability range.

The key idea is:

A predicted probability should have a meaningful statistical interpretation.

Calibration vs Accuracy

Calibration and classification accuracy measure different things.

Accuracy asks:

How often did the model predict the correct class?

Calibration asks:

Do the predicted probabilities correspond to the observed frequencies?

Consider two models.

Model A
Accuracy: 92%
Calibration: Poor

Model B
Accuracy: 90%
Calibration: Good

Model A isn’t necessarily better for every application.

If you need trustworthy probabilities, Model B may be more useful.

Why Calibration Matters

Probability estimates are particularly important when predictions drive decisions.

Examples include:

• Credit risk
• Fraud detection
• Customer churn
• Medical risk prediction
• Insurance
• Marketing
• Demand forecasting
• Resource allocation
• Ranking systems

Suppose a company wants to contact customers whose churn probability exceeds 70%.

If the model is poorly calibrated, the threshold may not represent the intended level of risk.

A calibrated model makes probability thresholds more meaningful.

A Simple Example

Imagine a fraud model predicts:

Transaction A → 0.90
Transaction B → 0.80
Transaction C → 0.70
Transaction D → 0.20
Transaction E → 0.10

These numbers represent predicted probabilities.

If the model is calibrated, transactions receiving a probability of approximately 0.80 should actually be fraudulent about 80% of the time over many similar predictions.

Calibration therefore gives probabilities practical meaning.

Calibration in Classification

Calibration is primarily relevant when a classifier produces probability estimates.

For binary classification, a model might produce:

P(y = 1 | X) = 0.73

This means the model estimates a 73% probability that the observation belongs to class 1.

For multiclass classification, the model might produce:

Class A → 0.10
Class B → 0.70
Class C → 0.20

The probabilities should sum to approximately 1.

Calibration methods can also be applied to multiclass classification, although evaluation requires additional considerations.

Step 1: Train a Classification Model

Let’s start with a simple binary classification example.

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier

X, y = make_classification(
    n_samples=5000,
    n_features=10,
    random_state=42
)

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42,
    stratify=y
)

model = RandomForestClassifier(
    random_state=42
)

model.fit(X_train, y_train)

The model can now produce class probabilities.

probabilities = model.predict_proba(X_test)[:, 1]

Step 2: Inspect Predicted Probabilities

You can examine the first few predictions:

print(probabilities[:10])

You might see:

[0.82, 0.14, 0.76, 0.43, 0.91, ...]

These numbers are the model’s estimated probabilities for the positive class.

But we still don’t know whether they are well calibrated.

Step 3: Plot a Calibration Curve

A calibration curve compares predicted probabilities with observed frequencies.

Scikit-learn provides:

from sklearn.calibration import calibration_curve

fraction_of_positives, mean_predicted_value = calibration_curve(
    y_test,
    probabilities,
    n_bins=10
)

You can visualize the result:

import matplotlib.pyplot as plt

plt.plot(
    mean_predicted_value,
    fraction_of_positives,
    marker="o"
)

plt.plot(
    [0, 1],
    [0, 1],
    linestyle="--"
)

plt.xlabel("Mean predicted probability")
plt.ylabel("Fraction of positives")
plt.title("Calibration Curve")
plt.show()

The diagonal represents perfect calibration.

The closer the model’s curve is to that diagonal, the better calibrated its probabilities are.

Understanding the Calibration Curve

Suppose the model produces:

Predicted Probability	Actual Positive Rate
0.10	0.08
0.30	0.27
0.50	0.48
0.70	0.61
0.90	0.78

The model is increasingly overconfident at higher probabilities.

For example:

Predicted: 90%
Observed: 78%

The model is predicting probabilities that are too high.

The Perfect Calibration Line

The reference line represents:

Predicted probability = Observed frequency

For example:

Predicted 0.20 → Actual 0.20
Predicted 0.50 → Actual 0.50
Predicted 0.80 → Actual 0.80

A perfectly calibrated model would follow this relationship.

Common Calibration Methods

Two common approaches are:

Sigmoid Calibration

Also called Platt scaling, sigmoid calibration learns a logistic transformation of the model’s scores.

It is useful when the relationship between the classifier’s scores and actual probabilities can be reasonably represented by a smooth sigmoid function.

In scikit-learn:

CalibratedClassifierCV(
    model,
    method="sigmoid"
)

Isotonic Calibration

Isotonic regression learns a non-decreasing mapping between the original predictions and observed probabilities.

It is more flexible than sigmoid calibration.

In scikit-learn:

CalibratedClassifierCV(
    model,
    method="isotonic"
)

However, isotonic regression generally needs more calibration data to avoid overfitting.

Step 4: Calibrate With Sigmoid Scaling

The simplest approach is to use CalibratedClassifierCV.

from sklearn.calibration import CalibratedClassifierCV

calibrated_model = CalibratedClassifierCV(
    model,
    method="sigmoid",
    cv=5
)

calibrated_model.fit(X_train, y_train)

Then generate calibrated probabilities:

calibrated_probabilities = calibrated_model.predict_proba(
    X_test
)[:, 1]

Now compare these probabilities with the original model.

Step 5: Calibrate With Isotonic Regression

You can change the method:

calibrated_model = CalibratedClassifierCV(
    model,
    method="isotonic",
    cv=5
)

calibrated_model.fit(X_train, y_train)

calibrated_probabilities = calibrated_model.predict_proba(
    X_test
)[:, 1]

The rest of the workflow remains similar.

Sigmoid vs Isotonic Calibration

Method	Flexibility	Data Requirement	Risk
Sigmoid	Lower	Lower	Less likely to overfit
Isotonic	Higher	Higher	Can overfit with limited data

A useful practical rule is:

Start with sigmoid calibration when the calibration dataset is relatively small.

Consider isotonic calibration when you have enough calibration data and the relationship appears more complex.

Why Cross-Validation Matters

You should avoid calibrating a model using exactly the same predictions that were used to train it.

Otherwise, the calibration process can learn from overly optimistic predictions.

CalibratedClassifierCV uses cross-validation to help separate model fitting from calibration.

Conceptually:

Training Data
     ↓
Cross-Validation
     ↓
Model Predictions
     ↓
Calibration
     ↓
Calibrated Model

This helps reduce information leakage.

Calibration Data vs Training Data

A clean setup separates the roles of the datasets.

For example:

Training Set
    ↓
Train Model

Calibration Set
    ↓
Learn Probability Mapping

Test Set
    ↓
Final Evaluation

The test set should remain untouched until the final evaluation.

Cross-validation can automate parts of this process.

Evaluating Calibration With Brier Score

The Brier score measures the difference between predicted probabilities and actual binary outcomes.

For binary classification, lower values are better.

Scikit-learn provides:

from sklearn.metrics import brier_score_loss

score = brier_score_loss(
    y_test,
    calibrated_probabilities
)

print(score)

You can compare the original and calibrated models:

original_score = brier_score_loss(
    y_test,
    probabilities
)

calibrated_score = brier_score_loss(
    y_test,
    calibrated_probabilities
)

print("Original:", original_score)
print("Calibrated:", calibrated_score)

A lower Brier score generally indicates better probabilistic accuracy.

Calibration Is Not Just About Brier Score

A model can have a reasonable Brier score without being perfectly calibrated.

Therefore, use multiple evaluation methods.

Useful tools include:

• Calibration curves
• Brier score
• Log loss
• Reliability diagrams
• Expected calibration error
• Discrimination metrics such as ROC AUC

These measure different aspects of model performance.

Calibration vs Discrimination

This distinction is extremely important.

Discrimination asks whether the model can distinguish positive cases from negative cases.

Metrics include:

ROC AUC
PR AUC

Calibration asks whether the predicted probabilities correspond to actual frequencies.

A model can therefore have:

Excellent discrimination
Poor calibration

or:

Good calibration
Moderate discrimination

These are not contradictory.

Does Calibration Change Accuracy?

Calibration changes probability estimates.

It doesn’t necessarily improve classification accuracy.

Suppose you classify:

probability >= 0.5 → positive
probability < 0.5 → negative

Calibration may move probabilities around without changing many class predictions.

Therefore, don’t calibrate a model solely because you want higher accuracy.

Calibrate when trustworthy probabilities matter.

Calibration and Classification Thresholds

Calibration becomes especially useful when decisions depend on probability thresholds.

Suppose:

Probability > 0.80
→ Send transaction for manual review

If the model is poorly calibrated, 0.80 may not actually represent an 80% risk.

After calibration, probability thresholds become easier to interpret.

Calibration for Imbalanced Classification

Calibration can become more complicated with highly imbalanced datasets.

Suppose only:

1%

of transactions are fraudulent.

A model could predict low probabilities for most observations and still appear well behaved on some aggregate metrics.

You should evaluate:

• Calibration
• Precision
• Recall
• PR AUC
• Class prevalence

Calibration should always be evaluated in the context of the actual problem.

Calibration and Random Forests

Tree-based models can produce poorly calibrated probabilities in some settings.

Random forests, gradient boosting models, and other powerful classifiers may produce useful rankings while their probability estimates need calibration.

That doesn’t mean these models are bad.

It means:

Prediction Ranking
        ≠
Probability Calibration

Calibration can improve the usefulness of their probability outputs.

Calibration and Logistic Regression

Logistic regression often produces comparatively well-behaved probabilities when its assumptions are appropriate.

But this doesn’t mean it is automatically perfectly calibrated.

You should still evaluate calibration when probability quality matters.

Calibration and Neural Networks

Modern neural networks can also be poorly calibrated.

In some cases, neural networks may become overconfident.

Calibration methods can help adjust their probability outputs after training.

Calibration for Multiclass Models

Suppose your model predicts:

Cat     0.10
Dog     0.75
Rabbit  0.15

For multiclass problems, calibration is more complex because each class has its own probability.

You can still use:

CalibratedClassifierCV(
    model,
    method="sigmoid",
    cv=5
)

and evaluate the resulting probabilities using suitable multiclass metrics such as log loss.

Calibration With a Pipeline

If preprocessing is required, combine it with the classifier using a pipeline.

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.calibration import CalibratedClassifierCV

base_model = make_pipeline(
    StandardScaler(),
    LogisticRegression()
)

calibrated_model = CalibratedClassifierCV(
    base_model,
    method="sigmoid",
    cv=5
)

calibrated_model.fit(X_train, y_train)

This ensures preprocessing occurs within the cross-validation process rather than leaking information across folds.

Comparing Original and Calibrated Models

A useful evaluation workflow is:

from sklearn.metrics import (
    brier_score_loss,
    log_loss,
    roc_auc_score
)

original_prob = model.predict_proba(X_test)[:, 1]
calibrated_prob = calibrated_model.predict_proba(X_test)[:, 1]

print(
    "Original Brier:",
    brier_score_loss(y_test, original_prob)
)

print(
    "Calibrated Brier:",
    brier_score_loss(y_test, calibrated_prob)
)

print(
    "Original Log Loss:",
    log_loss(y_test, original_prob)
)

print(
    "Calibrated Log Loss:",
    log_loss(y_test, calibrated_prob)
)

print(
    "Original ROC AUC:",
    roc_auc_score(y_test, original_prob)
)

print(
    "Calibrated ROC AUC:",
    roc_auc_score(y_test, calibrated_prob)
)

This lets you determine whether calibration improved probabilistic performance without assuming that it did.

Plotting Before and After Calibration

You can compare calibration curves:

from sklearn.calibration import calibration_curve
import matplotlib.pyplot as plt

prob_true_original, prob_pred_original = calibration_curve(
    y_test,
    original_prob,
    n_bins=10
)

prob_true_calibrated, prob_pred_calibrated = calibration_curve(
    y_test,
    calibrated_prob,
    n_bins=10
)

plt.plot(
    prob_pred_original,
    prob_true_original,
    marker="o",
    label="Original"
)

plt.plot(
    prob_pred_calibrated,
    prob_true_calibrated,
    marker="o",
    label="Calibrated"
)

plt.plot(
    [0, 1],
    [0, 1],
    linestyle="--",
    label="Perfect calibration"
)

plt.xlabel("Mean predicted probability")
plt.ylabel("Fraction of positives")
plt.legend()
plt.show()

This gives you a visual comparison of probability reliability.

Expected Calibration Error

Another commonly discussed measure is Expected Calibration Error (ECE).

ECE divides predictions into probability bins and compares:

Average predicted probability

with:

Observed positive frequency

A simplified concept is:

ECE
≈
Weighted average of
|predicted probability - observed frequency|

Lower values indicate closer agreement.

However, ECE depends on implementation choices such as the number and type of bins, so it should not be treated as the only calibration metric.

Common Calibration Mistakes

Calibrating on the Test Set

Don’t use the test set to learn the calibration mapping.

The test set should be reserved for final evaluation.

Optimizing Only for Accuracy

Calibration is about probability quality, not simply class predictions.

Ignoring Sample Size

Flexible calibration methods such as isotonic regression can overfit when calibration data is limited.

Assuming Every Model Is Poorly Calibrated

Always measure calibration before deciding that calibration is necessary.

Looking Only at One Metric

Use calibration curves alongside numerical metrics.

Ignoring Class Imbalance

Calibration should be evaluated against realistic class prevalence.

Applying Calibration After Data Leakage

If preprocessing uses information from validation or test data, calibration results can become misleading.

Best Practices

Measure Before Calibrating

Establish a baseline first.

Use Cross-Validation

Avoid learning the calibration mapping from the same predictions used to fit the model.

Keep the Test Set Untouched

Use it only for final evaluation.

Start With Sigmoid Calibration

It is often a sensible baseline when calibration data is limited.

Consider Isotonic Regression With More Data

Its flexibility can be useful when the calibration relationship isn’t well represented by a sigmoid.

Compare Probability Metrics

Use metrics such as Brier score and log loss.

Inspect Calibration Curves

Visual diagnostics can reveal problems that a single number hides.

Evaluate the Business Decision

A small numerical improvement may not matter if it doesn’t improve the decision process.

A Practical Calibration Workflow

A reusable workflow looks like:

Raw Data
    ↓
Train/Test Split
    ↓
Train Classifier
    ↓
Generate Probabilities
    ↓
Measure Calibration
    ↓
Choose Calibration Method
    ↓
Calibrate
    ↓
Evaluate
    ↓
Select Model
    ↓
Deploy
    ↓
Monitor Calibration

Importantly, calibration doesn’t necessarily end when the model is deployed.

Real-world data distributions can change.

A model that was well calibrated during development may become poorly calibrated later.

Monitoring Calibration in Production

After deployment, monitor:

• Predicted probability distributions
• Actual outcome rates
• Calibration curves
• Brier score
• Log loss
• Data drift
• Class prevalence

For example:

Production Model

Predicted risk:
0.70

Observed outcome rate:
0.51

If this pattern persists, the model may have become miscalibrated.

This can happen because of:

• Population changes
• Behavior changes
• New products
• Policy changes
• Data pipeline changes
• Concept drift

When Should You Calibrate a Model?

Calibration is especially valuable when:

• Probabilities drive decisions
• Risk thresholds matter
• Costs vary by prediction
• Multiple models are compared using probabilities
• Predictions are used for resource allocation
• Probability estimates are communicated to users

It may be less important when you only need a ranking or a hard class prediction.

Conclusion

Classification models don’t just produce predictions.

They often produce probabilities.

Those probabilities can be extremely useful—but only when they are reasonably trustworthy.

Model calibration aligns predicted probabilities with observed outcome frequencies.

In Python, scikit-learn makes calibration relatively straightforward with:

CalibratedClassifierCV

The two common approaches are:

• Sigmoid calibration for a simpler, more constrained probability mapping
• Isotonic calibration for a more flexible, non-decreasing mapping when sufficient calibration data is available

The most important lesson is that calibration is different from accuracy.

A model can correctly rank observations while producing probabilities that are too high or too low.

If your application depends on statements such as:

“This customer has a 70% probability of churning.”

then you should ask whether that 70% actually corresponds to reality.

That is the purpose of calibration.

Frequently Asked Questions