Handling Missing Data in Python: 5 Proven Techniques

Messy datasets are a data analyst’s worst nightmare and missing values are usually the main culprit. Whether you’re working with survey data, financial records, or machine learning datasets, missing data can lead to biased results, model errors, or misleading insights.

Fortunately, Python (especially with the pandas library) offers several powerful ways to detect, analyze, and fix missing data. In this guide, you’ll learn 5 proven techniques for handling missing values — from simple fixes to advanced strategies.

What Is Missing Data?

Missing data occurs when no value is stored for a variable in an observation. It can happen due to:

• Human error (e.g., skipped survey questions)
• Data corruption or incomplete imports
• Sensor or API failures
• Merging datasets with mismatched keys

Step 1: Import Libraries

import pandas as pd
import numpy as np

df = pd.read_csv("data.csv")
print(df.isnull().sum())

This helps identify which columns have missing values.

Technique 1: Drop Missing Values

When data is small or the missing percentage is low, dropping rows/columns may be best.

df.dropna(inplace=True)

To drop only specific columns:

df.dropna(subset=['Age', 'Salary'], inplace=True)

Use when: less than 5% of total data is missing.

Technique 2: Fill Missing Values with Mean/Median/Mode

df['Age'].fillna(df['Age'].mean(), inplace=True)
df['Salary'].fillna(df['Salary'].median(), inplace=True)
df['City'].fillna(df['City'].mode()[0], inplace=True)

This keeps dataset size consistent while reducing bias.

Technique 3: Forward/Backward Fill

Useful in time-series or sequential data.

df.fillna(method='ffill', inplace=True)  # Forward fill
df.fillna(method='bfill', inplace=True)  # Backward fill

Real-world example: Filling missing stock prices using previous day’s data.

Technique 4: Conditional Imputation

Fill missing data based on other values.

df.loc[df['Gender'] == 'Male', 'Income'] = df.loc[df['Gender'] == 'Male', 'Income'].fillna(df.loc[df['Gender'] == 'Male', 'Income'].mean())

This accounts for subgroup differences, improving accuracy.

Technique 5: Predictive Imputation (Using Machine Learning)

For advanced projects, use models to estimate missing values.

from sklearn.linear_model import LinearRegression

model = LinearRegression()
train = df[df['Salary'].notnull()]
test = df[df['Salary'].isnull()]

model.fit(train[['Age', 'Experience']], train['Salary'])
df.loc[df['Salary'].isnull(), 'Salary'] = model.predict(test[['Age', 'Experience']])

This is best for large datasets where missing values are significant.

Visualizing Missing Data

import seaborn as sns
import matplotlib.pyplot as plt

sns.heatmap(df.isnull(), cbar=False, cmap='viridis')
plt.title("Missing Data Visualization")
plt.show()

Seeing your missing patterns helps you decide which technique works best.

Best Practices

• Always check percentage of missing data before deciding.
• Avoid dropping large portions of data.
• Keep a backup copy of your raw dataset.
• Document every imputation method for reproducibility.

Handling missing data is a crucial step in every data project.
Using these 5 proven techniques, you can restore dataset integrity, improve analysis accuracy, and ensure your machine learning models make reliable predictions.

Explore more hands-on data tutorials at CodeWithFimi.com

FAQs