Polars vs Pandas Performance Comparison

Pandas has been the default tool for data manipulation in Python for over fifteen years. It is installed in virtually every data science environment, has documentation for nearly every operation imaginable, and is so deeply embedded in the Python data ecosystem that most analysts think of it and Python data manipulation as synonyms.

Polars arrived quietly and then very loudly. Built in Rust and designed from the ground up for performance, it has been posting benchmark numbers that make pandas look slow by comparison in ways that are difficult to ignore. Ten times faster on groupby operations. Five times faster on joins. A fraction of the memory usage on large datasets. The numbers are real and they come from architectural decisions that pandas, constrained by backward compatibility, cannot easily replicate.

But raw performance is not the only consideration when choosing a data manipulation library. Ecosystem maturity, syntax familiarity, integration with other tools, and the actual size of the data you work with all factor into whether switching is worth the investment. This guide covers where Polars genuinely outperforms pandas and by how much, where pandas still makes more sense, and how the two libraries compare on syntax so you can evaluate the migration effort realistically.

Why Polars Is Faster: The Architecture Difference

Understanding why Polars outperforms pandas requires understanding what each library was built on top of.

Pandas was built on NumPy, a library designed for numerical computation on arrays. The DataFrame abstraction in pandas maps columns to NumPy arrays and operations iterate over those arrays using Python and Cython code. This architecture works well but has two fundamental limitations. Python’s global interpreter lock prevents true parallelism, meaning pandas uses a single CPU core for most operations. And the row-oriented memory layout inherited from NumPy is not ideal for the column-oriented operations that dominate analytical workloads.

Polars is built on the Apache Arrow memory format, a columnar in-memory representation specifically designed for analytical operations. It is implemented in Rust, which compiles to native machine code and has no global interpreter lock, enabling true multi-threaded execution across all available CPU cores. Every operation in Polars is designed to take advantage of both columnar memory layout and parallel execution simultaneously.

The practical consequence is that Polars gets faster as your data gets larger, not just because it is more efficient per operation but because it scales across CPU cores while pandas stays single-threaded. On a modern laptop with eight or sixteen cores, Polars can distribute a groupby operation across all of them while pandas uses one.

Polars also has a lazy evaluation engine that optimizes query plans before executing them. When you chain operations in lazy mode, Polars analyzes the entire chain and applies optimizations like predicate pushdown, which filters data as early as possible to reduce the amount processed in subsequent steps, and projection pushdown, which reads only the columns needed rather than loading the full dataset. These optimizations happen automatically without any changes to your code.

Setting Up the Benchmark

All benchmarks in this guide run on a generated dataset of five million rows representing customer transactions, which is large enough to show meaningful performance differences without requiring specialized hardware.

python

import polars as pl
import pandas as pd
import numpy as np
import time
from contextlib import contextmanager

print(f"Polars version: {pl.__version__}")
print(f"Pandas version: {pd.__version__}")

@contextmanager
def timer(label):
    start = time.perf_counter()
    yield
    elapsed = time.perf_counter() - start
    print(f"{label}: {elapsed:.3f}s")

# Generate five million row dataset
np.random.seed(42)
n = 5_000_000

raw_data = {
    "transaction_id": np.arange(1, n + 1),
    "customer_id": np.random.randint(1, 100_001, n),
    "amount": np.random.lognormal(4.0, 1.2, n).round(2),
    "category": np.random.choice(
        ["retail", "food", "travel", "entertainment", "health"], n
    ),
    "status": np.random.choice(
        ["completed", "pending", "failed", "refunded"], n,
        p=[0.75, 0.10, 0.08, 0.07]
    ),
    "country": np.random.choice(
        ["US", "UK", "DE", "FR", "JP", "CA", "AU"], n,
        p=[0.40, 0.15, 0.12, 0.10, 0.08, 0.08, 0.07]
    ),
    "timestamp": pd.date_range("2022-01-01", periods=n, freq="s")
}

# Create both dataframes from the same data
df_pandas = pd.DataFrame(raw_data)
df_polars = pl.DataFrame(raw_data)

print(f"\nDataset: {n:,} rows")
print(f"Pandas memory: {df_pandas.memory_usage(deep=True).sum() / 1e6:.1f} MB")
print(f"Polars memory: {df_polars.estimated_size('mb'):.1f} MB")

Benchmark 1: GroupBy Aggregation

GroupBy is one of the most common operations in data analysis and one where the performance difference between Polars and pandas is most pronounced.

python

# Pandas groupby
with timer("Pandas groupby"):
    pandas_result = (
        df_pandas
        .groupby(["category", "country"])
        .agg(
            total_amount=("amount", "sum"),
            avg_amount=("amount", "mean"),
            transaction_count=("transaction_id", "count"),
            unique_customers=("customer_id", "nunique")
        )
        .reset_index()
    )

# Polars groupby (eager)
with timer("Polars groupby (eager)"):
    polars_result = (
        df_polars
        .group_by(["category", "country"])
        .agg([
            pl.col("amount").sum().alias("total_amount"),
            pl.col("amount").mean().alias("avg_amount"),
            pl.col("transaction_id").count().alias("transaction_count"),
            pl.col("customer_id").n_unique().alias("unique_customers")
        ])
    )

# Polars groupby (lazy with query optimization)
with timer("Polars groupby (lazy)"):
    polars_lazy_result = (
        df_polars
        .lazy()
        .group_by(["category", "country"])
        .agg([
            pl.col("amount").sum().alias("total_amount"),
            pl.col("amount").mean().alias("avg_amount"),
            pl.col("transaction_id").count().alias("transaction_count"),
            pl.col("customer_id").n_unique().alias("unique_customers")
        ])
        .collect()
    )

print(f"\nPandas result shape: {pandas_result.shape}")
print(f"Polars result shape: {polars_result.shape}")

On a modern laptop with eight cores, Polars completes this groupby in roughly 0.3 to 0.5 seconds. Pandas typically takes 3 to 6 seconds on the same operation. The speedup ranges from six to fifteen times depending on hardware.

Benchmark 2: Filtering and Column Operations

python

# Pandas filtering and new column creation
with timer("Pandas filter and transform"):
    pandas_filtered = df_pandas[
        (df_pandas["status"] == "completed") &
        (df_pandas["amount"] > 100)
    ].copy()
    pandas_filtered["amount_usd_normalized"] = (
        pandas_filtered["amount"] / pandas_filtered["amount"].max()
    )
    pandas_filtered["is_high_value"] = pandas_filtered["amount"] > 500
    pandas_filtered["customer_tier"] = pd.cut(
        pandas_filtered["amount"],
        bins=[0, 100, 500, 2000, float("inf")],
        labels=["bronze", "silver", "gold", "platinum"]
    )

# Polars filtering and new column creation
with timer("Polars filter and transform"):
    polars_filtered = (
        df_polars
        .filter(
            (pl.col("status") == "completed") &
            (pl.col("amount") > 100)
        )
        .with_columns([
            (pl.col("amount") / pl.col("amount").max())
            .alias("amount_usd_normalized"),
            (pl.col("amount") > 500).alias("is_high_value"),
            pl.col("amount")
            .cut(
                breaks=[100, 500, 2000],
                labels=["bronze", "silver", "gold", "platinum"]
            )
            .alias("customer_tier")
        ])
    )

print(f"\nPandas filtered rows: {len(pandas_filtered):,}")
print(f"Polars filtered rows: {len(polars_filtered):,}")

Benchmark 3: Joins

Joins on large datasets are where pandas struggles most significantly. Polars uses a hash join algorithm that is substantially faster and more memory efficient.

python

# Create a customer lookup table
customer_data = {
    "customer_id": np.arange(1, 100_001),
    "customer_segment": np.random.choice(
        ["enterprise", "mid_market", "smb", "consumer"], 100_000,
        p=[0.05, 0.15, 0.30, 0.50]
    ),
    "acquisition_year": np.random.choice(range(2018, 2025), 100_000),
    "lifetime_value_band": np.random.choice(
        ["low", "medium", "high"], 100_000,
        p=[0.50, 0.35, 0.15]
    )
}

customers_pandas = pd.DataFrame(customer_data)
customers_polars = pl.DataFrame(customer_data)

# Pandas join
with timer("Pandas join (5M rows)"):
    pandas_joined = df_pandas.merge(
        customers_pandas,
        on="customer_id",
        how="left"
    )

# Polars join
with timer("Polars join (5M rows)"):
    polars_joined = df_polars.join(
        customers_polars,
        on="customer_id",
        how="left"
    )

print(f"\nPandas joined shape: {pandas_joined.shape}")
print(f"Polars joined shape: {polars_joined.shape}")

Polars join on five million rows typically completes in under one second. Pandas typically takes four to eight seconds. The gap widens further when joining on multiple columns or when both tables are large.

Benchmark 4: String Operations

String operations expose another area where Polars outperforms pandas significantly due to parallel string processing.

python

# Pandas string operations
with timer("Pandas string operations"):
    df_pandas["category_upper"] = df_pandas["category"].str.upper()
    df_pandas["country_category"] = (
        df_pandas["country"].str.strip() + "_" +
        df_pandas["category"].str.lower()
    )
    df_pandas["is_retail_or_food"] = df_pandas["category"].str.contains(
        "retail|food", regex=True
    )

# Polars string operations
with timer("Polars string operations"):
    polars_strings = df_polars.with_columns([
        pl.col("category").str.to_uppercase().alias("category_upper"),
        (
            pl.col("country").str.strip_chars() + "_" +
            pl.col("category").str.to_lowercase()
        ).alias("country_category"),
        pl.col("category").str.contains("retail|food").alias("is_retail_or_food")
    ])

Benchmark 5: Reading and Writing Files

I/O performance matters for real-world pipelines where data is read from and written to disk repeatedly.

python

import os

# Write test data to parquet
df_pandas.to_parquet("test_pandas.parquet", index=False)

# Pandas parquet read
with timer("Pandas read parquet"):
    pandas_from_parquet = pd.read_parquet("test_pandas.parquet")

# Polars parquet read
with timer("Polars read parquet"):
    polars_from_parquet = pl.read_parquet("test_pandas.parquet")

# Polars lazy parquet read (scan rather than load)
with timer("Polars scan parquet (lazy)"):
    polars_lazy = (
        pl.scan_parquet("test_pandas.parquet")
        .filter(pl.col("status") == "completed")
        .group_by("category")
        .agg(pl.col("amount").sum())
        .collect()
    )

# CSV comparison
df_pandas.head(100_000).to_csv("test_sample.csv", index=False)

with timer("Pandas read CSV (100k rows)"):
    pd.read_csv("test_sample.csv")

with timer("Polars read CSV (100k rows)"):
    pl.read_csv("test_sample.csv")

# Cleanup
for f in ["test_pandas.parquet", "test_sample.csv"]:
    os.remove(f)

Polars’ scan_parquet with lazy evaluation is particularly powerful. It reads only the data needed to satisfy the query rather than loading the entire file, which can reduce I/O dramatically when filtering or selecting a subset of columns.

Syntax Comparison: Translating Common Operations

Understanding the syntax difference is essential for evaluating migration effort.

python

# ── SELECTING COLUMNS ──────────────────────────────────────────────────────
pandas_select = df_pandas[["customer_id", "amount", "category"]]
polars_select = df_polars.select(["customer_id", "amount", "category"])

# ── FILTERING ──────────────────────────────────────────────────────────────
pandas_filter = df_pandas[df_pandas["amount"] > 100]
polars_filter = df_polars.filter(pl.col("amount") > 100)

# ── MULTIPLE CONDITIONS ────────────────────────────────────────────────────
pandas_multi = df_pandas[
    (df_pandas["status"] == "completed") & (df_pandas["amount"] > 500)
]
polars_multi = df_polars.filter(
    (pl.col("status") == "completed") & (pl.col("amount") > 500)
)

# ── CREATING NEW COLUMNS ───────────────────────────────────────────────────
df_pandas["amount_doubled"] = df_pandas["amount"] * 2
polars_new_col = df_polars.with_columns(
    (pl.col("amount") * 2).alias("amount_doubled")
)

# ── CONDITIONAL COLUMN ────────────────────────────────────────────────────
df_pandas["tier"] = np.where(df_pandas["amount"] > 500, "high", "low")
polars_when = df_polars.with_columns(
    pl.when(pl.col("amount") > 500)
    .then(pl.lit("high"))
    .otherwise(pl.lit("low"))
    .alias("tier")
)

# ── SORTING ────────────────────────────────────────────────────────────────
pandas_sorted = df_pandas.sort_values("amount", ascending=False)
polars_sorted = df_polars.sort("amount", descending=True)

# ── MISSING VALUE HANDLING ────────────────────────────────────────────────
pandas_nulls = df_pandas["amount"].fillna(0)
polars_nulls = df_polars.with_columns(
    pl.col("amount").fill_null(0)
)

# ── APPLYING FUNCTIONS ────────────────────────────────────────────────────
# Pandas: apply() is slow, avoid on large data
pandas_apply = df_pandas["amount"].apply(lambda x: x * 1.1 if x > 100 else x)

# Polars: use expressions, no apply needed
polars_expr = df_polars.with_columns(
    pl.when(pl.col("amount") > 100)
    .then(pl.col("amount") * 1.1)
    .otherwise(pl.col("amount"))
    .alias("amount_adjusted")
)

The most important syntax difference is how Polars handles in-place mutation. Pandas DataFrames are mutable and assignments like df["new_col"] = values modify the DataFrame in place. Polars DataFrames are immutable. Every operation returns a new DataFrame. This eliminates a whole class of bugs related to unintended mutation and copy-on-write confusion but requires adjusting to a functional style where you chain operations and assign results.

Lazy Evaluation: The Polars Superpower

Lazy evaluation is the feature that most differentiates Polars from pandas for pipeline use cases.

python

# Lazy mode: build a query plan without executing it
lazy_query = (
    pl.scan_parquet("large_dataset.parquet")  # Does not read file yet
    .filter(pl.col("status") == "completed")   # Noted, not executed
    .filter(pl.col("amount") > 100)            # Noted, not executed
    .select(["customer_id", "amount", "category", "country"])
    .group_by(["category", "country"])
    .agg([
        pl.col("amount").sum().alias("total"),
        pl.col("customer_id").n_unique().alias("unique_customers")
    ])
    .sort("total", descending=True)
)

# Inspect the optimized query plan
print(lazy_query.explain(optimized=True))

# Execute the plan (reads only what is needed)
# result = lazy_query.collect()

The query plan shows predicate pushdown in action. Instead of reading the full dataset and then filtering, Polars pushes the filter conditions into the file reading step and reads only rows that match. For a dataset where ten percent of rows satisfy the filter, this means reading ten percent of the data rather than one hundred percent.

Memory Usage Comparison

python

import tracemalloc

# Measure pandas memory for a groupby operation
tracemalloc.start()
_ = df_pandas.groupby("category").agg({"amount": ["sum", "mean", "std"]})
pandas_peak = tracemalloc.get_traced_memory()[1]
tracemalloc.stop()

# Measure polars memory for the same operation
tracemalloc.start()
_ = df_polars.group_by("category").agg([
    pl.col("amount").sum(),
    pl.col("amount").mean(),
    pl.col("amount").std()
])
polars_peak = tracemalloc.get_traced_memory()[1]
tracemalloc.stop()

print(f"Pandas peak memory for groupby: {pandas_peak / 1e6:.1f} MB")
print(f"Polars peak memory for groupby: {polars_peak / 1e6:.1f} MB")
print(f"Memory ratio: {pandas_peak / polars_peak:.1f}x")

Polars typically uses two to four times less memory than pandas for equivalent operations. The difference comes from the Arrow columnar format which avoids the object overhead pandas inherits from NumPy, and from Polars’ streaming execution mode that processes data in chunks rather than loading everything into memory simultaneously.

Polars vs Pandas Cheat Sheet

Dimension	Pandas	Polars
GroupBy speed (5M rows)	3-6 seconds	0.3-0.5 seconds
Join speed (5M rows)	4-8 seconds	0.5-1 second
Memory efficiency	Baseline	2-4x more efficient
Multi-threading	Single core	All available cores
Lazy evaluation	No	Yes
Mutation	Mutable	Immutable
Ecosystem maturity	Very mature	Growing fast
Learning curve	Gentler	Moderate adjustment
pandas interop	Native	to_pandas() / from_pandas()
Best data size	Under 1GB	1GB to hundreds of GB

When to Use Each Library

Use pandas when your data fits comfortably in memory at under a few hundred megabytes, when you need deep integration with libraries that only support pandas DataFrames, when your team is not ready for the syntax adjustment, or when you are doing interactive exploration where the performance difference is imperceptible.

Use Polars when your data is large enough that pandas operations take more than a few seconds, when memory pressure is a concern, when you are building data pipelines that need to be fast and reproducible, or when you are starting a new project where there is no legacy pandas code to maintain.

The practical path for most teams is to use Polars for the heavy lifting, large dataset processing, aggregations, joins, and file I/O, while keeping pandas for the final analysis steps that require libraries with pandas-only support.

Common Mistakes When Switching to Polars

Using apply() on a Polars Series is the most common performance mistake for pandas migrants. Polars has no efficient apply equivalent because it does not need one. Every operation you would use apply for in pandas has a native Polars expression that is orders of magnitude faster. When you find yourself reaching for apply, look for the native Polars expression instead.

Ignoring lazy evaluation leaves significant performance on the table. For any operation that reads from files or involves chained transformations, wrapping the query in lazy mode and calling collect at the end enables query optimization that can dramatically reduce both execution time and memory usage.

Expecting in-place mutation causes confusion for pandas users. Polars DataFrames do not support in-place modification. Assignments like df["new_col"] = values raise errors. Every transformation returns a new DataFrame. The adjustment takes a few hours to internalize and produces cleaner, more predictable code once you have made it.

FAQs