06 - Lazy API and Performance - Python

28 min read

Lazy API and Performance - Python

Quote

“Premature optimization is the root of all evil.”

— Donald Knuth, Structured Programming with go to Statements (1974)

“The First Rule of Program Optimization: Don’t do it. The Second Rule of Program Optimization (for experts only): Don’t do it yet.”

— Michael A. Jackson, Principles of Program Design (1975)

Runs the live example for this step.

import pandas as pd
import polars as pl
import polars.selectors as cs
import numpy as np
from pathlib import Path
 
from IPython.display import display, Markdown
html_formatter = get_ipython().display_formatter.formatters['text/html'] # type: ignore
html_formatter.for_type(pd.DataFrame, lambda df: df.to_html())
html_formatter.for_type(pd.Series, lambda s: s.to_frame().to_html())
 
pl.Config.set_tbl_rows(100)
pd.set_option("display.max_rows", 100)
 
DATA = Path("../data")
 
# Core datasets
ohlcv_pd = pd.read_parquet(DATA / "eurostoxx50_ohlcv.parquet")  # 66K rows, daily OHLCV
ohlcv_pl = pl.read_parquet(DATA / "eurostoxx50_ohlcv.parquet")
dim_pd = pd.read_parquet(DATA / "index_dim.parquet")            # 169 rows, stock metadata
dim_pl = pl.read_parquet(DATA / "index_dim.parquet")
scores_pd = pd.read_parquet(DATA / "scores_daily.parquet")      # 466 rows, composite scores
scores_pl = pl.read_parquet(DATA / "scores_daily.parquet")
 
print(f"OHLCV: {ohlcv_pd.shape}, Dim: {dim_pd.shape}, Scores: {scores_pd.shape}")
import time

OHLCV: (66355, 12), Dim: (169, 26), Scores: (466, 36)

Eager: Immediate

Eager File Read

Polars | Eager read with read_parquet()

Eager execution loads ALL data

Eager execution loads ALL data into memory immediately pl.read_parquet() and pd.read_parquet() load the entire file into RAM. For files larger than available memory, use lazy mode (pl.scan_parquet()) or Pandas read_parquet(columns=[...]) to only load needed columns.

Use lazy scanning or column selection to limit memory usage

In Polars, replace pl.read_parquet(path) with pl.scan_parquet(path) and chain .select() / .filter() before .collect() — Polars will apply projection and predicate pushdown automatically. In Pandas, pass columns=[...] to pd.read_parquet() to load only the columns you need.

Reads eurostoxx50_ohlcv.parquet eagerly with pl.read_parquet(), printing the resulting type and shape to confirm all 66,355 rows are fully loaded into a Polars DataFrame.

df = pl.read_parquet(DATA / "eurostoxx50_ohlcv.parquet")
print(f"Type: {type(df)}, Shape: {df.shape}")

Type: <class ‘polars.dataframe.frame.DataFrame’>, Shape: (66355, 12)

Lazy: Deferred

The lazy-vs-eager distinction mirrors concepts elsewhere in the pipeline: dbt’s ephemeral models defer computation in the same way a LazyFrame does, while dbt run materializes results like .collect() — see dbt-materializations. BigQuery’s query planner applies similar predicate pushdown and projection pruning, covered in querying-and-cost-optimization.


flowchart LR
    A["pl.scan_parquet(path)<br/>pl.scan_csv(path)"] -->|"LazyFrame<br/>(no data)"| B["Build query plan"]
    B --> C[".filter(pl.col(...))"]
    C --> D[".select(...)"]
    D --> E[".group_by / .sort<br/>.with_columns"]
    E -->|"Optimizer rewrites plan"| F[".collect()"]
    F --> G["DataFrame<br/>(materialized)"]
    B2["pd.read_parquet(path)"] -->|"DataFrame<br/>(all data loaded)"| G2["Immediate execution<br/>(no optimization)"]

    style A fill:#1f2335,stroke:#7aa2f7,color:#c0caf5
    style B fill:#1f2335,stroke:#7aa2f7,color:#c0caf5
    style C fill:#1f2335,stroke:#e0af68,color:#c0caf5
    style D fill:#1f2335,stroke:#e0af68,color:#c0caf5
    style E fill:#1f2335,stroke:#e0af68,color:#c0caf5
    style F fill:#1f2335,stroke:#f7768e,color:#f7768e
    style G fill:#1f2335,stroke:#9ece6a,color:#9ece6a
    style B2 fill:#1f2335,stroke:#f7768e,color:#c0caf5
    style G2 fill:#1f2335,stroke:#565f89,color:#c0caf5

Lazy File Scanning

Polars | scan_parquet()

pl.scan_parquet(path) returns a LazyFrame — a description of work, not data. Call .collect_schema() to inspect the column names and types without reading any rows. The schema is derived from the Parquet file footer metadata.

Scans eurostoxx50_ohlcv.parquet without loading any rows, confirms the resulting type is LazyFrame, then calls collect_schema() to read the Parquet footer and return all 12 column names with their Arrow types.

lf = pl.scan_parquet(DATA / "eurostoxx50_ohlcv.parquet")
print(f"Type: {type(lf)}")
print(f"Schema: {lf.collect_schema()}")

Type: <class ‘polars.lazyframe.frame.LazyFrame’> Schema: Schema({‘id’: Int64, ‘symbol’: String, ‘date’: Date, ‘open’: Float64, ‘high’: Float64, ‘low’: Float64, ‘close’: Float64, ‘adj_close’: Float64, ‘volume’: Int64, ‘dividends’: Float64, ‘stock_splits’: Float64, ‘is_filled’: Boolean})

Collect

Materializing a LazyFrame

Polars | .collect()

Lazy chains do nothing until .collect()

Forgetting .collect() is the most common Polars mistake A LazyFrame does nothing until .collect() is called. If you assign lf.filter(...) to a variable and never collect, no computation happens. Unlike Pandas (where every operation runs immediately), Polars lazy chains must end with .collect() to materialize results.

Always end a Polars lazy chain with .collect()

Every pl.scan_*() chain must terminate with .collect() to produce a DataFrame. Use type annotations (lf: pl.LazyFrame, df: pl.DataFrame) to catch missing .collect() calls at review time. If you need a partial result during development, chain .head(100).collect() first to verify the plan before collecting the full dataset.

Chains filter, select, sort, and head on a lazy scan of eurostoxx50_ohlcv.parquet, then materializes with .collect() to return the 10 most recent closing prices for ASML.AS.

result = (
    pl.scan_parquet(DATA / "eurostoxx50_ohlcv.parquet")
    .filter(pl.col("symbol") == "ASML.AS")
    .select("symbol", "date", "close")
    .sort("date", descending=True)
    .head(10)
    .collect()
)
display(result)
symboldateclose
strdatef64
ASML.AS2026-03-121190.8
ASML.AS2026-03-111198.8
ASML.AS2026-03-101200.0
ASML.AS2026-03-091147.6
ASML.AS2026-03-061147.0
ASML.AS2026-03-051186.0
ASML.AS2026-03-041199.8
ASML.AS2026-03-031161.8
ASML.AS2026-03-021210.4
ASML.AS2026-02-271233.4

Query Plan

Query Plan Inspection

Polars | explain()

lf.explain() returns the optimized query plan as a string without executing it. Read it to verify that Polars has applied predicate and projection pushdown. The key fields to look for:

  • PROJECT N/12 COLUMNS — only N columns will be read from disk (projection pushdown active)
  • SELECTION: [...] — the filter predicate has been pushed into the scanner
  • ESTIMATED ROWS — Polars’ row count estimate before execution

Always check .explain() before collecting on large datasets

On a multi-GB Parquet file, call lf.explain() first to verify that PROJECT shows fewer columns than the total and that SELECTION contains your filter. If you see PROJECT */N COLUMNS with the full column count, your filter or select is not pushing down — check for unsupported expression types.

Builds a lazy plan filtering ASML.AS rows with close above 900 and selecting 3 of 12 columns, then calls explain() to print the optimized plan — confirming PROJECT 3/12 COLUMNS and both filter predicates merged into a single SELECTION clause.

lf = (
    pl.scan_parquet(DATA / "eurostoxx50_ohlcv.parquet")
    .filter(pl.col("symbol") == "ASML.AS")
    .filter(pl.col("close") > 900)
    .select("symbol", "date", "close")
)
print(lf.explain())

Parquet SCAN [../data/eurostoxx50_ohlcv.parquet] PROJECT 3/12 COLUMNS SELECTION: [([(col(“symbol”)) == (“ASML.AS”)]) & ([(col(“close”)) > (900.0)])] ESTIMATED ROWS: 66355

Predicate Pushdown

Predicate Pushdown

Polars | Predicate pushdown

When you call .filter() on a LazyFrame, Polars moves the predicate into the file scanner at optimization time. For Parquet files, the scanner uses row group statistics to skip entire row groups that cannot satisfy the predicate — so unmatched rows are never deserialized into memory. The optimizer applies this even if you write the filter after a .select().

Pandas has no predicate pushdown

pd.read_parquet() loads all rows unconditionally. The only way to limit rows in Pandas is to read all data first, then filter with df.query() or boolean indexing. To limit I/O in Pandas, use pd.read_parquet(path, filters=[...]) which delegates pushdown to the pyarrow engine — but this is only available at read time, not as part of a chain.

Chains .select() before .filter() to demonstrate that the optimizer still pushes the symbol predicate into the Parquet scanner regardless of chain order — confirmed by explain() showing SELECTION: [(col("symbol")) == ("ASML.AS")].

lf = (
    pl.scan_parquet(DATA / "eurostoxx50_ohlcv.parquet")
    .select("symbol", "date", "close")
    .filter(pl.col("symbol") == "ASML.AS")
)
print("Filter pushed to scan:")
print(lf.explain())

Filter pushed to scan: Parquet SCAN [../data/eurostoxx50_ohlcv.parquet] PROJECT 3/12 COLUMNS SELECTION: [(col(“symbol”)) == (“ASML.AS”)] ESTIMATED ROWS: 66355

Projection Pushdown

Projection Pushdown

Polars | Projection pushdown

.select() on a LazyFrame tells Polars which columns are needed. At optimization time, the column list is pushed into the Parquet scanner, which reads only those byte ranges from disk — all other columns are completely skipped. The .explain() output shows PROJECT N/12 COLUMNS to confirm this is active.

Selects 2 of 12 columns from the OHLCV Parquet file via a lazy scan, calls explain() to confirm PROJECT 2/12 COLUMNS is active, then collects to verify the resulting shape is (66355, 2).

lf = pl.scan_parquet(DATA / "eurostoxx50_ohlcv.parquet").select("symbol", "close")
print("Only 2 columns read:")
print(lf.explain())
print(f"Result: {lf.collect().shape}")

Only 2 columns read: Parquet SCAN [../data/eurostoxx50_ohlcv.parquet] PROJECT 2/12 COLUMNS ESTIMATED ROWS: 66355 Result: (66355, 2)

Eager to Lazy Conversion

Converting Eager to Lazy

Polars | .lazy()

Call .lazy() on an existing DataFrame to enter the lazy API. The conversion is free — no data is copied. Use this when you have already read data eagerly but want to apply further operations with query optimization before collecting. pl.col("name") is the expression API entry point — it references a column by name and is the foundation for all Polars filter, select, and transform expressions.

Reads the OHLCV dataset eagerly, then calls .lazy() to enter the lazy API and chains .filter() and .select() before .collect() — returning 1,331 ASML.AS rows with only the date and close columns.

df = pl.read_parquet(DATA / "eurostoxx50_ohlcv.parquet")
result = df.lazy().filter(pl.col("symbol") == "ASML.AS").select("date", "close").collect()
print(f"Result: {result.shape}")

Result: (1331, 2)

Streaming Mode

Streaming Execution

Polars | collect(engine=“streaming”)

Streaming mode processes data in chunks instead of loading the full dataset into memory at once. Pass engine="streaming" to .collect() to activate it. Use streaming for datasets larger than available RAM or when you want bounded memory usage on long-running aggregations.

Streaming support is still incomplete

Not all operations support streaming. Unsupported nodes fall back to in-memory execution silently. Check .explain(streaming=True) to see which plan nodes will stream. The new Polars streaming engine (introduced in v1) is more capable than the legacy streaming=True parameter from v0.x but remains under active development.

Confirm streaming in the plan before relying on it

Treat engine="streaming" as a bounded-memory candidate, not a guarantee. Run .explain(streaming=True) first and make sure the critical scan, filter, and aggregation nodes remain on the streaming path before you depend on it in production.

Scans the OHLCV Parquet with engine="streaming", filters rows where close exceeds 500, groups by symbol to compute average close rounded to 2 decimals, sorts descending, and collects — returning the 7 symbols that consistently traded above 500.

result = (
    pl.scan_parquet(DATA / "eurostoxx50_ohlcv.parquet")
    .filter(pl.col("close") > 500)
    .group_by("symbol").agg(pl.col("close").mean().round(2).alias("avg_close"))
    .sort("avg_close", descending=True)
    .collect(engine="streaming")
)
display(result.head(10))
symbolavg_close
strf64
RMS.PA1761.56
ADYEN.AS1545.98
RHM.DE1230.09
ASML.AS696.35
MC.PA677.03
ARGX.BR625.29
MUV2.DE548.06

Profile

Query Profiling

Polars | .profile()

lf.profile() executes the query and returns a tuple of (result_df, timing_df). The timing_df contains one row per plan node with start and end timestamps in microseconds. Use it to identify which operation in a chain is the bottleneck before optimizing.

Profiles a chain that filters two symbols (ASML.AS and MC.PA), computes daily return percentage via with_columns, and groups by symbol for average return — returning both the result DataFrame and a timing DataFrame with microsecond start/end timestamps per plan node.

lf = (
    pl.scan_parquet(DATA / "eurostoxx50_ohlcv.parquet")
    .filter(pl.col("symbol").is_in(["ASML.AS", "MC.PA"]))
    .with_columns(((pl.col("close") - pl.col("open")) / pl.col("open") * 100).alias("ret"))
    .group_by("symbol").agg(pl.col("ret").mean().round(4).alias("avg_ret"))
)
result_df, timing_df = lf.profile()
display(result_df)
display(timing_df)
symbolavg_ret
strf64
ASML.AS-0.0162
MC.PA-0.0045
nodestartend
stru64u64
optimization02054
with_column(ret)20542222
group_by(symbol)22272566

The start and end columns are in microseconds. Here, optimization took 2054 µs (plan rewriting), with_column(ret) took 168 µs, and group_by took 339 µs. Subtract end - start per row to find the slowest node — that is where to focus optimization effort.

Pandas vs Polars Lazy Benchmark

Head Query Benchmark

Pandas / Polars | Head query benchmark

Compares the same operation — filter one symbol, select three columns, sort by date descending, take top 10 — using Pandas eager execution vs Polars lazy execution. Pandas reads the full Parquet file then filters in memory. Polars scans with predicate and projection pushdown.

When to choose Polars lazy over Pandas for read queries?

For small DataFrames (< 100K rows, fits easily in memory), the difference is negligible and Pandas’ familiar API may be preferable. Choose Polars lazy when: (1) data is larger than memory or growing toward that limit, (2) the query reads from Parquet and you can exploit projection/predicate pushdown, (3) the operation is part of a scheduled pipeline where throughput matters, or (4) you need reproducible multi-threaded performance.

Times the identical head-10 query — filter ASML.AS, select 3 columns, sort descending by date — in Pandas (full eager read then filter) versus Polars lazy scanning with predicate and projection pushdown, printing both durations and the speedup ratio (5.1x in the sample run).

start = time.perf_counter()
pd.read_parquet(DATA / "eurostoxx50_ohlcv.parquet").query("symbol == 'ASML.AS'")[["symbol","date","close"]].sort_values("date", ascending=False).head(10)
pd_t = time.perf_counter() - start
start = time.perf_counter()
pl.scan_parquet(DATA / "eurostoxx50_ohlcv.parquet").filter(pl.col("symbol")=="ASML.AS").select("symbol","date","close").sort("date", descending=True).head(10).collect()
pl_t = time.perf_counter() - start
print(f"Pandas: {pd_t:.4f}s \nPolars lazy: {pl_t:.4f}s \nSpeedup: {pd_t/pl_t:.1f}x")

Pandas: 0.0114s Polars lazy: 0.0023s Speedup: 5.1x

Summary

FeaturePolars EagerPolars LazyPandas
ExecutionImmediate.collect()Immediate
OptimizationNonePushdownNone
MemoryFullStreamingFull
Query planNo.explain()No
ProfileNo.profile()No

Performance & Optimization

Dataset Reload

Reloads both datasets into memory so the benchmark cells below have a clean baseline without any cached filtered subsets from earlier cells.

Runs the live example for this step.

ohlcv_pd=pd.read_parquet(DATA/"eurostoxx50_ohlcv.parquet")
ohlcv_pl=pl.read_parquet(DATA/"eurostoxx50_ohlcv.parquet")
print(f"Rows: {len(ohlcv_pd):,}")

Rows: 66,355

Vectorized vs Loop

Row Iteration vs Vectorized Arithmetic

Pandas | iterrows() vs column arithmetic

DataFrame.iterrows() yields one Python dict per row, bypassing NumPy’s C-level vectorization entirely. Column arithmetic (df["a"] - df["b"]) dispatches to NumPy’s C implementation and processes all rows in a single SIMD-accelerated pass. The benchmark below measures iterrows over 1K rows vs column subtraction over the full 66K rows.

Polars has no iterrows equivalent

Polars DataFrames are immutable and expression-based — there is no row iteration API. All operations use pl.col() expressions that execute in parallel across the full column in Rust. This design eliminates the anti-pattern at the API level.

Runs iterrows() over 1,000 rows computing close minus open per row in Python, then runs direct column subtraction over all 66,355 rows, printing both durations to quantify the cost of per-row Python dispatch versus a single C-level vectorized pass.

start=time.perf_counter()
results=[]
for _,row in ohlcv_pd.head(1000).iterrows():
    results.append(row["close"]-row["open"])
bad=time.perf_counter()-start
 
start=time.perf_counter()
_=ohlcv_pd["close"]-ohlcv_pd["open"]
good=time.perf_counter()-start
 
print(f"iterrows (1K): {bad:.4f}s")
print(f"vectorized (66K): {good:.4f}s")

iterrows (1K): 0.0116s vectorized (66K): 0.0003s

Why apply() Is Slow

The apply() Anti-Pattern

Pandas | apply(axis=1) anti-pattern

apply() is extremely slow

df.apply(axis=1) is 100-1000x slower than vectorized operations. apply() with axis=1 iterates row by row in Python — bypassing NumPy/C optimizations entirely. The example below shows a 743x speedup from vectorization. Every apply(lambda r: ...) in production code is a performance bug. Rewrite using column arithmetic, .where(), or np.select() for conditional logic.

Replace apply(axis=1) with vectorized column arithmetic

Rewrite row-wise lambdas as direct column operations: df["ret"] = (df["close"] - df["open"]) / df["open"] * 100. For conditional logic use np.where() or np.select() instead of apply. In Polars, use pl.when().then().otherwise() — all operations execute in parallel across columns in native Rust with no Python overhead.

Applies a row-wise lambda computing daily return percentage (close−open)/open×100 over all 66K rows with apply(axis=1), then computes the same metric via direct column arithmetic, printing both durations and the 743x speedup.

start=time.perf_counter()
ohlcv_pd["ret_apply"]=ohlcv_pd.apply(lambda r:(r["close"]-r["open"])/r["open"]*100,axis=1)
apply_t=time.perf_counter()-start
 
start=time.perf_counter()
ohlcv_pd["ret_vec"]=(ohlcv_pd["close"]-ohlcv_pd["open"])/ohlcv_pd["open"]*100
vec_t=time.perf_counter()-start
 
print(f"apply: {apply_t:.4f}s\nvectorized: {vec_t:.4f}s\nSpeedup: {apply_t/vec_t:.0f}x")

apply: 0.2317s vectorized: 0.0003s Speedup: 743x

Memory Usage

RAM Footprint Comparison

Pandas / Polars | RAM footprint comparison

Polars uses Apache Arrow as its in-memory format. Arrow stores data in typed, contiguous column buffers that are more compact than Pandas’ NumPy arrays, which add per-array Python object overhead and use 8-byte floats for integer columns that contain NaN.

Pandas uses NaN (float) for missing integers; Polars uses null

In Pandas, an integer column with any missing value is silently promoted to float64 to accommodate NaN. This doubles the memory footprint for integer columns with nulls and can cause silent precision loss for large integers. Polars uses a native null type backed by a validity bitmask — integer columns stay as Int64 regardless of nulls, with no type coercion.

Measures the in-memory size of the 66K-row OHLCV dataset in both libraries — memory_usage(deep=True) for Pandas and estimated_size("mb") for Polars — printing both in MB and the ratio to confirm Polars is 2.2x more memory-efficient.

mem_pd=ohlcv_pd.memory_usage(deep=True).sum()/1024/1024
mem_pl=ohlcv_pl.estimated_size("mb")
print(f"Pandas: {mem_pd:.2f} MB")
print(f"Polars: {mem_pl:.2f} MB")
print(f"Ratio: {mem_pd/mem_pl:.1f}x")

Pandas: 11.65 MB Polars: 5.20 MB Ratio: 2.2x

Benchmark: Common Operations

Filter, GroupBy, and Sort Benchmarks

Pandas / Polars | Filter, GroupBy, Sort

Measures three core operations — single-column equality filter, group-by mean aggregation, and multi-column sort — side by side on the same 66K-row dataset. Polars benefits from multi-threaded execution and Apache Arrow’s cache-friendly columnar layout.

Benchmarks filter, group-by mean aggregation, and multi-column sort on the 66K-row OHLCV dataset with Pandas versus Polars, each in a separate timed code cell, printing per-operation durations and speedup ratios.

ops = {}
 
start = time.perf_counter()
_ = ohlcv_pd[ohlcv_pd["symbol"] == "ASML.AS"]
ops["pd_filter"] = time.perf_counter() - start
 
start = time.perf_counter()
_ = ohlcv_pl.filter(pl.col("symbol") == "ASML.AS")
ops["pl_filter"] = time.perf_counter() - start
 
print(f"  pd_filter : {ops['pd_filter']:.4f}s")
print(f"  pl_filter : {ops['pl_filter']:.4f}s")
print(f"  Filter speedup: {ops['pd_filter'] / ops['pl_filter']:.1f}x")

pd_filter : 0.0019s pl_filter : 0.0006s Filter speedup: 3.4x

Runs the live example for this step.

start = time.perf_counter()
_ = ohlcv_pd.groupby("symbol")["close"].mean()
ops["pd_groupby"] = time.perf_counter() - start
 
start = time.perf_counter()
_ = ohlcv_pl.group_by("symbol").agg(pl.col("close").mean())
ops["pl_groupby"] = time.perf_counter() - start
 
print(f"  pd_groupby : {ops['pd_groupby']:.4f}s")
print(f"  pl_groupby : {ops['pl_groupby']:.4f}s")
print(f"  GroupBy speedup: {ops['pd_groupby'] / ops['pl_groupby']:.1f}x")

pd_groupby : 0.0025s pl_groupby : 0.0013s GroupBy speedup: 1.9x

Runs the live example for this step.

start = time.perf_counter()
_ = ohlcv_pd.sort_values(["symbol", "date"])
ops["pd_sort"] = time.perf_counter() - start
 
start = time.perf_counter()
_ = ohlcv_pl.sort("symbol", "date")
ops["pl_sort"] = time.perf_counter() - start
 
print(f"  pd_sort : {ops['pd_sort']:.4f}s")
print(f"  pl_sort : {ops['pl_sort']:.4f}s")
print(f"  Sort speedup: {ops['pd_sort'] / ops['pl_sort']:.1f}x")

pd_sort : 0.0064s pl_sort : 0.0014s Sort speedup: 4.6x

Polars Architecture

Execution Model

Polars | Execution model

Polars is built on four pillars that together make it faster than Pandas for analytical workloads:

  • Apache Arrow — columnar in-memory format; cache-friendly, zero-copy between Arrow-native systems (DuckDB, Pyarrow, Pandas 2.x Arrow backend)
  • SIMD vector instructions — operations on column arrays use CPU vectorization (AVX2/AVX-512) to process multiple values per clock cycle
  • Multi-threaded Rust engine — the thread pool size matches available CPU cores; group-by and sort operations partition work across threads automatically
  • Lazy query optimization — the full pipeline is rewritten before execution: predicate pushdown, projection pushdown, common subexpression elimination

Pandas and Polars have no shared index

Pandas attaches an index to every DataFrame. The index enables label-based alignment in joins and assignments but is also a source of subtle bugs (misaligned index after filtering, accidental index-based join instead of column-based). Polars has no index — every operation is explicit and column-based. This makes Polars code more predictable but means you must use explicit join keys rather than relying on index alignment.

Runs the live example for this step.

print(f"Thread pool: {pl.thread_pool_size()}")
print(f"Polars version: {pl.__version__}")

Thread pool: 16 Polars version: 1.39.3

Summary

AspectPandasPolars
EngineSingle-threaded CMulti-threaded Rust
MemoryNumPy arraysApache Arrow
OptimizationNoneQuery planning
apply()SlowUse expressions

Common Traps and Safe Patterns

Collect Once Per Workload

Calling .collect() inside a loop re-runs the full plan

If a loop calls lf.filter(...).collect() on each iteration, the entire read and filter pipeline runs from scratch every time. That turns lazy execution into repeated full-query work.

Build one lazy plan and collect once

Combine conditions into a single expression tree, then materialize the final result once. If you need several variants, keep the shared lazy base and branch it deliberately instead of collecting on every loop pass.

Benchmark at Production Scale

Toy benchmarks hide the real performance gap

A 1,000-row benchmark may show Pandas and Polars performing identically. At 1M+ rows, Polars’ multi-threaded Rust engine and query optimization often change the result completely.

Benchmark with realistic row counts and schema width

Measure on data sizes, column counts, and value distributions that look like production. That is the only way to judge whether lazy execution, threading, and pushdown are paying off.

Measure String Columns Deeply

Shallow memory accounting understates object-column cost

Pandas df.memory_usage() counts only pointer sizes for object columns. For strings, the shallow estimate can miss most of the real Python-object memory.

Use deep=True when inspecting Pandas memory

Call df.memory_usage(deep=True) in Pandas and compare it with df.estimated_size() in Polars. That gives you a usable memory baseline before deciding whether the pipeline still fits comfortably in RAM.

Keep Computation Out of Python Row Loops

.apply() throws away vectorized execution

A Python lambda applied row by row runs at Python speed. Once you move work into .apply(axis=1) or similar callbacks, you give up the C/Rust execution path that makes DataFrame code fast.

Rewrite row logic as native expressions

Replace row-wise lambdas with column arithmetic, conditional expressions, or other native APIs. Treat Python UDFs as a temporary escape hatch, not the steady-state design.

Verify the Streaming Path

Streaming requests can still materialize the whole dataset

If a query contains an unsupported operation, Polars may fall back to a non-streaming plan. Assuming that engine="streaming" is always honored is a memory-footgun.

Check explain(streaming=True) before trusting bounded memory

Inspect the plan and confirm the important scan, filter, and aggregation nodes stay on the streaming path. If they do not, reduce the query or accept that it will run as a normal in-memory collect.

Python Lazy API and Performance Recommendations

  1. Use lazy for anything that touches disk — scan_parquet(), scan_csv(), scan_ndjson() enable predicate and projection pushdown that eager reads cannot match.
  2. Inspect query plans — call .explain() on every production query to verify that pushdown and fusion are active. If the plan shows a full scan where you expected pushdown, the filter expression may be too complex.
  3. Benchmark at realistic scale — test with 1M+ rows, realistic column counts, and representative data distributions. Toy benchmarks mislead.
  4. Profile before optimizing — use lf.profile() to identify the actual bottleneck. Optimizing the wrong step wastes effort.
  5. Replace every .apply() with a native expression — treat .apply() / .map_elements() as a temporary workaround, not a solution. Almost every Python lambda has a vectorized equivalent.
  6. Measure memory with deep=True — always use df.memory_usage(deep=True) (Pandas) or df.estimated_size() (Polars) to get accurate memory figures.
  7. Build with eager, ship with lazy — explore interactively using eager DataFrames, then convert the final pipeline to lazy for production.

Troubleshooting and failure modes

Failure modes

lf.explain() still shows a full scan

When .explain() still reports PROJECT */12 COLUMNS or leaves the filter outside the scan node, the query is still reading too much data. Rewrite the filter with native expressions, move it before joins or aggregations, and verify pushdown on the next plan inspection.

Inspect a lazy Parquet scan that should push the filter into the scanner.

import polars as pl
from pathlib import Path
 
DATA = Path("C:/Users/aperi/My Drive/VAULT/data")
lf = (
    pl.scan_parquet(DATA / "eurostoxx50_ohlcv.parquet")
    .filter(pl.col("symbol") == "ASML.AS")
    .select("symbol", "close")
)
print("\n".join(lf.explain().splitlines()[:3]))
Parquet SCAN [C:/Users/aperi/My Drive/VAULT/data/eurostoxx50_ohlcv.parquet]
PROJECT 2/12 COLUMNS
SELECTION: [(col("symbol")) == ("ASML.AS")]

.collect() raises memory pressure

When .collect() fails or crawls because the result does not fit cleanly in RAM, keep the lazy pipeline intact and try a supported streaming collect before widening the scan again. If the plan cannot stream, reduce rows or columns earlier in the chain.

Collect the same scan in streaming mode to confirm the bounded-memory path still succeeds.

import polars as pl
from pathlib import Path
 
DATA = Path("C:/Users/aperi/My Drive/VAULT/data")
print(
    pl.scan_parquet(DATA / "eurostoxx50_ohlcv.parquet")
    .select("symbol", "close")
    .collect(engine="streaming")
    .shape
)
(66355, 2)

Python UDFs in filter() block pushdown

When a filter depends on a Python callback such as map_elements() or apply(), the optimizer cannot reason about it well enough to push it into the scan. Rewrite the predicate as a native expression on pl.col(...) so the scan node can stay optimizer-visible.

Show the native expression form that the optimizer can inspect and push down.

import polars as pl
 
expr = pl.col("close") > 900
print(f"Native expression: {expr}")
Native expression: [(col("close")) > (dyn int: 900)]

Toy benchmarks on 66,355 rows do not generalize

A benchmark that looks decisive on a single laptop-sized file can disappear at production scale, especially once I/O, cardinality, and wider schemas enter the picture. Measure on the largest realistic slice you can afford, not on a convenience sample.

Check the row count before trusting a speedup ratio from a small sample.

import pandas as pd
from pathlib import Path
 
DATA = Path("C:/Users/aperi/My Drive/VAULT/data")
ohlcv_pd = pd.read_parquet(DATA / "eurostoxx50_ohlcv.parquet")
print(f"Rows in sample: {ohlcv_pd.shape[0]:,}")
Rows in sample: 66,355

Streaming can still require explain(streaming=True)

If engine="streaming" does not behave as expected, inspect the plan rather than assuming the execution engine stayed on the chunked path. Unsupported operations can force a fallback, so verify the plan shape before relying on streaming for memory control.

Run a streaming-friendly aggregation to confirm the supported path produces a bounded result.

import polars as pl
from pathlib import Path
 
DATA = Path("C:/Users/aperi/My Drive/VAULT/data")
print(
    pl.scan_parquet(DATA / "eurostoxx50_ohlcv.parquet")
    .filter(pl.col("close") > 500)
    .group_by("symbol")
    .agg(pl.col("close").mean())
    .collect(engine="streaming")
    .shape
)
(7, 2)

lf.profile() points to the scan when I/O dominates

When the profile table shows optimization is cheap but scan or materialization costs remain large, the fastest win is usually narrower reads, better file layout, or faster storage. Use the timing rows to decide whether the bottleneck is planning, compute, or disk.

Profile a short lazy aggregation and inspect the timing rows that identify the slowest nodes.

import polars as pl
from pathlib import Path
 
DATA = Path("C:/Users/aperi/My Drive/VAULT/data")
lf = (
    pl.scan_parquet(DATA / "eurostoxx50_ohlcv.parquet")
    .filter(pl.col("symbol").is_in(["ASML.AS", "MC.PA"]))
    .with_columns(((pl.col("close") - pl.col("open")) / pl.col("open") * 100).alias("ret"))
    .group_by("symbol")
    .agg(pl.col("ret").mean().round(4).alias("avg_ret"))
)
_, timing_df = lf.profile()
print(timing_df)
shape: (3, 3)
┌──────────────────┬───────┬───────┐
│ node             ┆ start ┆ end   │
│ ---              ┆ ---   ┆ ---   │
│ str              ┆ u64   ┆ u64   │
╞══════════════════╪═══════╪═══════╡
│ optimization     ┆ 0     ┆ 9256  │
│ with_column(ret) ┆ 9256  ┆ 11262 │
│ group_by(symbol) ┆ 11267 ┆ 14176 │
└──────────────────┴───────┴───────┘

.apply(axis=1) is slow because it stays in Python

When a row-wise lambda still feels slow, replace it with a vectorized expression or a batch-oriented escape hatch like map_batches() if a native rewrite is not yet possible. The more you can keep inside columnar operations, the less Python overhead you pay.

Show the vectorized alternative that returns the same kind of per-row numeric differences without Python iteration.

import pandas as pd
from pathlib import Path
 
DATA = Path("C:/Users/aperi/My Drive/VAULT/data")
ohlcv_pd = pd.read_parquet(DATA / "eurostoxx50_ohlcv.parquet")
print((ohlcv_pd["close"] - ohlcv_pd["open"]).head(3).round(2).to_list())
[-0.94, 0.28, 0.81]

Graph View

Backlinks