How to Compare Machine Learning Models Fairly

How to Compare Machine Learning Models Fairly

Building a machine learning model is only one part of solving a data science problem. Another important task is determining whether the model performs better than other available options.

Imagine developing a model to predict whether a customer will cancel a subscription. You train three algorithms: Logistic Regression, Random Forest, and XGBoost. After evaluating them, you discover that XGBoost has the highest accuracy.

Does that automatically make XGBoost the best model?

Not necessarily.

The dataset may be imbalanced, the models may have been tested on different data splits, or one model may perform well on accuracy while producing more false negatives. You may also discover that the most accurate model is significantly slower, more expensive to deploy, or difficult to explain.

This is why machine learning models must be compared fairly.

A fair comparison ensures that every model is evaluated under consistent conditions, using appropriate metrics, the same test data, and a clearly defined objective. It helps data scientists avoid misleading conclusions and select models that solve the actual problem rather than simply producing impressive evaluation scores.

In this guide, you will learn how to compare machine learning models fairly, understand the evaluation metrics that matter, avoid common mistakes, and implement a practical model comparison workflow using Python.

1. What Does It Mean to Compare Machine Learning Models Fairly?

Comparing machine learning models fairly means evaluating different algorithms under the same experimental conditions so that differences in performance can be attributed to the models rather than inconsistencies in the evaluation process.

For example, suppose you want to compare:

  • Logistic Regression
  • Decision Tree
  • Random Forest
  • Support Vector Machine
  • XGBoost

A fair comparison requires you to give these models access to the same training data, evaluate them on the same unseen test data, and use consistent evaluation procedures.

If one model is trained on 80% of a dataset and another is trained on 60%, their results may not be directly comparable.

Similarly, evaluating one model using accuracy and another using F1-score does not provide a meaningful basis for ranking them.

A reliable comparison should consider:

  1. Data consistency.
  2. Appropriate preprocessing.
  3. Consistent training and validation procedures.
  4. Suitable evaluation metrics.
  5. Hyperparameter tuning.
  6. Model complexity and computational cost.
  7. Generalisation to unseen data.

The objective is not simply to identify the model with the highest score. It is to understand which model performs best for the problem being solved.

2. Start With a Clearly Defined Machine Learning Problem

Before comparing algorithms, establish what you want the models to achieve.

Different machine learning problems require different evaluation approaches.

For example:

ProblemExampleCommon evaluation metrics
ClassificationPredict customer churnPrecision, recall, F1-score, ROC-AUC
RegressionPredict house pricesMAE, MSE, RMSE, R²
ClusteringGroup customersSilhouette score, Davies-Bouldin index
Time series forecastingPredict monthly salesMAE, RMSE, MAPE
Anomaly detectionIdentify fraudulent transactionsPrecision, recall, PR-AUC

Consider a bank building a fraud detection model.

Its main objective may be to identify as many fraudulent transactions as possible without incorrectly blocking too many legitimate transactions.

In this situation, accuracy alone is not sufficient. A model that correctly classifies most legitimate transactions but misses a large proportion of fraudulent transactions could be unsuitable.

You should therefore define the business objective before selecting the evaluation metrics.

Ask questions such as:

  • What prediction are we trying to make?
  • What happens when the model makes an incorrect prediction?
  • Are false positives or false negatives more expensive?
  • How important is prediction speed?
  • Does the model need to be explainable?
  • How much computational power is available?

These questions help determine what a successful model actually looks like.

3. Use the Same Dataset for Every Model

One of the most important rules of fair model comparison is ensuring that every algorithm is evaluated using the same dataset.

Machine learning datasets are commonly divided into three groups:

Training dataset: Used to teach the model patterns and relationships in the data.

Validation dataset: Used to compare models, tune hyperparameters, and make development decisions.

Test dataset: Used to estimate how well the final selected model performs on previously unseen data.

A common split is:

  • 70% training data.
  • 15% validation data.
  • 15% test data.

Another common approach is an 80% training and 20% test split, with cross-validation performed on the training portion.

The appropriate split depends on dataset size, problem type, and data availability.

Why using the same test data matters

Imagine that you evaluate Logistic Regression on one test dataset and Random Forest on another.

The first dataset may contain more difficult examples, while the second may contain easier observations.

Even if Random Forest achieves a higher score, you cannot confidently conclude that it is better because the evaluation conditions were different.

Using the same test observations makes the comparison more meaningful.

Splitting data with Python

You can use Scikit-learn’s train_test_split() function to divide your dataset.

from sklearn.model_selection import train_test_split

X = df.drop("churn", axis=1)
y = df["churn"]

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42,
    stratify=y
)

The stratify=y argument helps preserve the proportion of target classes in both datasets. This is particularly useful when working with imbalanced classification datasets.

The random_state parameter makes the split reproducible.

For a fair comparison, use the same split for every model.

Important: Random splitting is not appropriate for every problem. Time series data should generally be split chronologically. Grouped observations, such as multiple records belonging to the same customer, may require group-aware splitting to prevent information leakage.

4. Apply Consistent Data Preprocessing

Data preprocessing can significantly influence machine learning performance.

Common preprocessing operations include:

  • Handling missing values.
  • Removing or investigating duplicate records.
  • Encoding categorical variables.
  • Scaling numerical features.
  • Handling outliers.
  • Selecting relevant features.

If different models receive differently processed versions of the dataset, their performance may not be directly comparable.

For example, Support Vector Machines and Logistic Regression can be sensitive to feature scales. A feature measured in thousands may dominate another feature measured between zero and one.

Decision Trees and Random Forests generally do not require feature scaling.

A fair approach is to use the same underlying information and apply appropriate preprocessing through reproducible pipelines.

Why Scikit-learn pipelines are useful

A pipeline ensures that preprocessing steps are consistently applied during training and prediction.

It also helps prevent data leakage because transformations are fitted using training data rather than the entire dataset.

For example:

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

logistic_model = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler()),
    ("model", LogisticRegression(max_iter=1000))
])

logistic_model.fit(X_train, y_train)

predictions = logistic_model.predict(X_test)

You can create separate pipelines for different algorithms while maintaining consistent preprocessing principles.

For example, you might include scaling for Logistic Regression and omit it for a Random Forest model.

The goal is not to force identical transformations on algorithms that have different requirements. The goal is to ensure that every model receives an equally valid opportunity to learn from the same information.

5. Choose the Right Evaluation Metrics

A model comparison is only as reliable as the metrics used to evaluate the models.

Different metrics measure different aspects of performance. Selecting the wrong metric can make an unsuitable model appear successful.

Classification metrics

Classification models predict categories such as fraud or legitimate, approved or rejected, and churn or no churn.

Common metrics include:

Accuracy

Accuracy measures the proportion of predictions the model gets correct.

Accuracy = Correct Predictions / Total Predictions

It is useful when classes are reasonably balanced and the costs of different errors are similar.

However, accuracy can be misleading when one class is significantly more common than another.

Precision

Precision measures how many positive predictions are actually positive.

Precision = True Positives / (True Positives + False Positives)

High precision is useful when false positives are expensive.

For example, a fraud detection system with low precision may incorrectly flag too many legitimate transactions.

Recall

Recall measures how many actual positive cases the model successfully identifies.

Recall = True Positives / (True Positives + False Negatives)

High recall is important when missing positive cases is costly, such as failing to identify fraudulent transactions.

F1-score

The F1-score is the harmonic mean of precision and recall.

F1-score = 2 × (Precision × Recall) / (Precision + Recall)

It provides a balance between precision and recall, although it does not directly account for true negatives.

ROC-AUC

The Area Under the Receiver Operating Characteristic Curve measures how well a classifier ranks positive examples above negative examples across different classification thresholds.

A higher ROC-AUC generally indicates stronger ranking performance.

For heavily imbalanced datasets, Precision-Recall AUC can be more informative because it focuses on performance involving the positive class.

Regression metrics

Regression models predict continuous numerical values, such as revenue, temperature, house prices, or customer spending.

Common metrics include:

Mean Absolute Error (MAE)

MAE calculates the average absolute difference between actual and predicted values.

MAE = Average of |Actual Value − Predicted Value|

It is easy to interpret and expressed in the same units as the target variable.

Mean Squared Error (MSE)

MSE calculates the average squared prediction error.

It penalises larger errors more heavily than smaller errors.

Root Mean Squared Error (RMSE)

RMSE is the square root of MSE.

It is expressed in the same units as the target variable and gives greater weight to large prediction errors.

R-squared (R²)

R² measures how much variation in the target variable is explained by the model relative to a baseline that predicts the target mean.

A higher value can indicate a better fit, but a high R² does not automatically mean a model will generalise well to new data.

Comparing common evaluation metrics

MetricBest used forMain consideration
AccuracyBalanced classificationCan hide minority-class errors
PrecisionLimiting false positivesMay reduce recall
RecallLimiting false negativesMay increase false positives
F1-scoreBalancing precision and recallDoes not directly include true negatives
ROC-AUCRanking binary classificationsCan look optimistic with severe imbalance
PR-AUCImbalanced classificationDepends on positive-class prevalence
MAERegressionTreats errors linearly
RMSERegressionPenalises large errors
R²RegressionDoes not measure absolute error

You should select metrics based on the actual problem rather than choosing whichever produces the highest score.

6. Use Cross-Validation to Reduce Evaluation Bias

A single train-test split can produce results that depend heavily on which observations end up in the training and testing datasets.

Cross-validation helps address this problem.

In k-fold cross-validation, the training dataset is divided into k approximately equal sections called folds.

The model is trained multiple times:

  1. The first fold is used for validation while the remaining folds are used for training.
  2. The second fold becomes the validation set.
  3. The process continues until every fold has been used for validation.
  4. The evaluation scores are combined.

For example, in five-fold cross-validation, each model is trained and evaluated five times using different training and validation combinations.

This provides a more reliable estimate of how a model performs across different subsets of the available data.

Implementing cross-validation with Python

from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier(
    n_estimators=100,
    random_state=42
)

scores = cross_val_score(
    model,
    X_train,
    y_train,
    cv=5,
    scoring="f1"
)

print("Fold scores:", scores)
print("Mean F1-score:", scores.mean())
print("Standard deviation:", scores.std())

The mean score provides an estimate of average performance, while the standard deviation indicates how much the results vary across folds.

A model with a slightly lower average score but much lower variability may be more reliable than one with a higher average and inconsistent results.

For classification tasks, use stratified cross-validation when preserving class proportions is important. For time series, use a time-aware validation strategy rather than ordinary shuffled folds.

7. Tune Hyperparameters Using the Same Evaluation Strategy

Hyperparameters are settings that control how machine learning algorithms learn.

Examples include:

  • Number of trees in a Random Forest.
  • Maximum tree depth.
  • Learning rate in gradient boosting.
  • Number of neighbours in K-Nearest Neighbours.
  • Regularisation strength in Logistic Regression.

Poorly selected hyperparameters can reduce model performance.

However, comparing one model after extensive tuning with another model using default settings is not necessarily fair.

Both models should receive a reasonable and documented tuning process.

Using GridSearchCV

Scikit-learn provides GridSearchCV, which evaluates different hyperparameter combinations using cross-validation.

from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier(random_state=42)

parameters = {
    "n_estimators": [100, 200, 300],
    "max_depth": [None, 10, 20],
    "min_samples_split": [2, 5]
}

search = GridSearchCV(
    estimator=model,
    param_grid=parameters,
    cv=5,
    scoring="f1",
    n_jobs=-1
)

search.fit(X_train, y_train)

print("Best parameters:", search.best_params_)
print("Best CV score:", search.best_score_)

You can apply a comparable tuning process to other algorithms.

For large search spaces, RandomizedSearchCV or more advanced optimisation methods may be more efficient than testing every possible combination.

Avoid test-set leakage: Do not repeatedly evaluate models on the final test set while changing hyperparameters. Use cross-validation on the training data for model selection, then evaluate the selected model on the test set.

8. Compare Model Performance Using Python

Now that you understand the main principles, let’s build a practical example comparing multiple classification algorithms.

We will use the Breast Cancer Wisconsin dataset available through Scikit-learn.

The objective is to classify observations as malignant or benign.

We will compare:

  • Logistic Regression.
  • Decision Tree.
  • Random Forest.
  • Support Vector Machine.

Step 1: Import the required libraries

import pandas as pd

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, cross_validate
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer

from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC

from sklearn.metrics import (
    accuracy_score,
    precision_score,
    recall_score,
    f1_score,
    roc_auc_score
)

Step 2: Load the dataset

data = load_breast_cancer()

X = pd.DataFrame(
    data.data,
    columns=data.feature_names
)

y = pd.Series(data.target)

print(X.head())
print(y.value_counts())

The dataset contains numerical measurements associated with breast cancer observations.

Scikit-learn encodes the target as:

  • 0: Malignant.
  • 1: Benign.

For this demonstration, we will treat benign as the positive class. In a real medical application, metric selection and the consequences of classification errors require careful clinical consideration.

Step 3: Split the dataset

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42,
    stratify=y
)

Every model will use the same training and test datasets.

Step 4: Define the models

models = {
    "Logistic Regression": Pipeline([
        ("imputer", SimpleImputer(strategy="median")),
        ("scaler", StandardScaler()),
        ("model", LogisticRegression(max_iter=2000))
    ]),

    "Decision Tree": Pipeline([
        ("imputer", SimpleImputer(strategy="median")),
        ("model", DecisionTreeClassifier(random_state=42))
    ]),

    "Random Forest": Pipeline([
        ("imputer", SimpleImputer(strategy="median")),
        ("model", RandomForestClassifier(
            n_estimators=200,
            random_state=42
        ))
    ]),

    "Support Vector Machine": Pipeline([
        ("imputer", SimpleImputer(strategy="median")),
        ("scaler", StandardScaler()),
        ("model", SVC(
            probability=True,
            random_state=42
        ))
    ])
}

Each model uses the same underlying dataset. Scaling is applied to algorithms that benefit from it, while tree-based models receive unscaled numerical features.

Step 5: Evaluate models using cross-validation

scoring = {
    "accuracy": "accuracy",
    "precision": "precision",
    "recall": "recall",
    "f1": "f1",
    "roc_auc": "roc_auc"
}

results = []

for name, model in models.items():

    scores = cross_validate(
        model,
        X_train,
        y_train,
        cv=5,
        scoring=scoring,
        n_jobs=-1
    )

    results.append({
        "Model": name,
        "Accuracy": scores["test_accuracy"].mean(),
        "Precision": scores["test_precision"].mean(),
        "Recall": scores["test_recall"].mean(),
        "F1 Score": scores["test_f1"].mean(),
        "ROC-AUC": scores["test_roc_auc"].mean()
    })

comparison = pd.DataFrame(results)

print(comparison.round(4))

This produces a table containing the average cross-validation performance of each algorithm.

The scores are based on the same folds and evaluation metrics, making the comparison more meaningful.

Step 6: Evaluate the selected models on the test dataset

After reviewing cross-validation results, select the most promising model based on your objective.

For demonstration, you can fit and evaluate all four models on the same held-out test dataset.

test_results = []

for name, model in models.items():

    model.fit(X_train, y_train)

    predictions = model.predict(X_test)

    probabilities = model.predict_proba(X_test)[:, 1]

    test_results.append({
        "Model": name,
        "Accuracy": accuracy_score(y_test, predictions),
        "Precision": precision_score(y_test, predictions),
        "Recall": recall_score(y_test, predictions),
        "F1 Score": f1_score(y_test, predictions),
        "ROC-AUC": roc_auc_score(y_test, probabilities)
    })

test_comparison = pd.DataFrame(test_results)

print(test_comparison.round(4))

The final table shows how each model performs on observations that were not used during training or cross-validation.

The results will depend on the dataset, software versions, and model settings. The important lesson is the evaluation process, not which algorithm happens to achieve the highest score.

9. Look Beyond Accuracy and Other Performance Scores

A model can achieve excellent evaluation scores and still be unsuitable for real-world deployment.

Fair model comparison should consider additional factors.

Model complexity

Some algorithms are easier to understand than others.

For example, a small Decision Tree can provide relatively straightforward decision rules, while a large ensemble of hundreds of trees may be harder to explain.

In regulated industries, interpretability may be an important requirement.

Training and prediction time

A model that takes several hours to train may be unsuitable for a project that requires frequent retraining.

Similarly, a model that takes several seconds to generate one prediction may not be appropriate for a real-time application.

Memory and computational requirements

Large models may require significant RAM, storage, or GPU resources.

These requirements can increase infrastructure costs and make deployment more difficult.

Interpretability

Businesses often need to understand why a model makes particular predictions.

For example, a financial institution may need to explain why a loan application was rejected.

A model with slightly lower predictive performance may be preferable if it provides clearer explanations and meets regulatory requirements.

Fairness across different groups

A model can perform well overall while producing significantly different error rates across demographic or operational groups.

Where relevant and lawful, evaluate metrics such as false positive rates, false negative rates, and recall across appropriate groups.

Fairness should be assessed using context-specific criteria rather than assuming that one metric guarantees an unbiased model.

10. Common Mistakes When Comparing Machine Learning Models

Even experienced data scientists can make mistakes that produce misleading evaluation results.

Here are some important ones to avoid.

1. Using different train-test splits

Comparing models evaluated on different test datasets introduces unnecessary variation.

Use the same appropriate split for every model.

2. Selecting models using accuracy alone

Accuracy can hide important classification errors, particularly when the target classes are imbalanced.

Choose metrics that reflect the problem’s objectives.

3. Data leakage

Data leakage happens when information from outside the training process influences model learning.

For example, scaling or imputing the entire dataset before splitting it can expose information from the test set.

Use pipelines and fit transformations only on the relevant training folds.

4. Giving one model more tuning effort

Comparing a heavily optimised model against another using default parameters may produce an unfair evaluation.

Use comparable tuning budgets and document the search process.

5. Repeatedly checking the test dataset

If you repeatedly change your model based on test results, you gradually adapt the model to the test set.

Keep the test dataset untouched until final evaluation.

6. Ignoring variability

A model’s score can change across different training samples.

Report cross-validation results, including mean and standard deviation, rather than relying on one score.

7. Ignoring real-world requirements

A model with the highest predictive score may be too expensive, slow, or difficult to maintain.

Include deployment requirements in your selection criteria.

11. A Practical Checklist for Fair Model Comparison

Before choosing a machine learning model, use the following checklist:

  • Define the prediction problem and business objective.
  • Select evaluation metrics that match the problem.
  • Use the same underlying dataset for all models.
  • Create appropriate training, validation, and test splits.
  • Prevent data leakage during preprocessing.
  • Use consistent cross-validation procedures.
  • Apply reasonable hyperparameter tuning.
  • Compare multiple performance metrics.
  • Evaluate model stability and variability.
  • Consider training time and prediction speed.
  • Examine interpretability and fairness requirements.
  • Evaluate the final selected model on an untouched test set.
  • Document the experiments and results.

Following this process makes your model selection decisions more transparent, reproducible, and defensible.

Conclusion

Comparing machine learning models fairly is about much more than identifying the algorithm with the highest accuracy.

A reliable comparison requires consistent datasets, appropriate preprocessing, suitable evaluation metrics, cross-validation, and a reasonable hyperparameter tuning process.

It also requires looking beyond predictive performance to consider factors such as interpretability, computational cost, fairness, and deployment requirements.

Python libraries such as Scikit-learn make it easier to implement these evaluation procedures and compare multiple algorithms within a consistent workflow.

Ultimately, the best machine learning model is not necessarily the most complex or the one with the highest score. It is the model that performs reliably on unseen data and meets the objectives of the problem it was designed to solve.

Frequently Asked Questions

1. What is the fairest way to compare machine learning models?

The fairest approach is to evaluate models using the same dataset, consistent data splits, appropriate preprocessing, identical cross-validation folds, and suitable evaluation metrics. You should also apply comparable hyperparameter tuning procedures.

2. Can I compare machine learning models using accuracy alone?

You can compare models using accuracy when the classification problem is reasonably balanced and different prediction errors have similar consequences. However, precision, recall, F1-score, ROC-AUC, and PR-AUC may provide more useful information depending on the problem.

3. Why is cross-validation important when comparing machine learning models?

Cross-validation evaluates models across multiple training and validation combinations. It reduces dependence on one particular data split and provides a better understanding of performance consistency.

4. Should every machine learning model use the same preprocessing steps?

Not necessarily. Different algorithms have different requirements. For example, Support Vector Machines and Logistic Regression often benefit from feature scaling, while Decision Trees generally do not require it. The important thing is to provide consistent and appropriate preprocessing without introducing data leakage.

5. How do I choose the best machine learning model?

Choose a model based on the problem’s objectives, evaluation metrics, generalisation performance, stability, interpretability, computational requirements, and deployment constraints. The highest-scoring model is not always the most suitable option.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top