01 - Foundations and I/O - Python

64 min read

Foundations and I/O - Python

Quote

“Bad programmers worry about the code. Good programmers worry about data structures and their relationships.”

— Linus Torvalds, Git mailing list post (2006)

import pandas as pd
import polars as pl
import numpy as np
from pathlib import Path
import time
 
from IPython.display import display, Markdown
html_formatter = get_ipython().display_formatter.formatters['text/html'] # type: ignore
html_formatter.for_type(pd.DataFrame, lambda df: df.to_html())
html_formatter.for_type(pd.Series, lambda s: s.to_frame().to_html())
 
pl.Config.set_tbl_rows(100)
pd.set_option("display.max_rows", 100)
 
DATA = Path("../data")
print(f"pandas  {pd.__version__}")
print(f"polars  {pl.__version__}")
print(f"numpy   {np.__version__}")
import shutil
import io
pandas  2.3.3
polars  1.39.3
numpy   2.4.3

Series

A Series is a one-dimensional labelled (Pandas) or unnamed (Polars) array. Pandas Series carry an index that enables label-based alignment; Polars Series carry only a name and rely on positional access.

Pandas vs Polars | Null representation

Pandas uses NaN (a float) for missing values, which silently promotes integer columns to float64. Polars uses a native Arrow null bitmask — the column dtype is preserved regardless of missing values. This is one of the most impactful behavioral differences between the two libraries.

Pandas / Polars | Creating a Series from a Python list

pd.Series() accepts a Python list and auto-generates a RangeIndex (0, 1, 2, …). The name parameter becomes the column header when the Series is placed in a DataFrame.

s_pd = pd.Series([10, 20, 30, 40], name="values")
print(type(s_pd))
display(s_pd)
<class 'pandas.core.series.Series'>
values
010
120
230
340

pl.Series(name, values) creates a Polars Series. Unlike Pandas, there is no index — only a name and positional data.

s_pl = pl.Series("values", [10, 20, 30, 40])
print(type(s_pl))
display(s_pl)
<class 'polars.series.series.Series'>
values
i64
10
20
30
40

Pandas / Polars | Creating a Series from a NumPy array

arr = np.array([1.1, 2.2, 3.3, np.nan, 5.5])
 
s_pd = pd.Series(arr, name="from_numpy")
display(s_pd)
print(f"dtype: {s_pd.dtype}")  # float64
from_numpy
01.1
12.2
23.3
3NaN
45.5
dtype: float64

Polars can wrap a NumPy array directly. Note that np.nan in NumPy is a float value, so Polars treats it as NaN (not null) when the source is a NumPy array.

s_pl = pl.Series("from_numpy", arr)
display(s_pl)
print(f"dtype: {s_pl.dtype}")  # Float64
from_numpy
f64
1.1
2.2
3.3
NaN
5.5
dtype: Float64

Pandas / Polars | Custom index vs named-only

Pandas supports a custom index on a Series — keys can be strings, dates, or any hashable type. This enables label-based access via .loc[].

s_pd = pd.Series(
    [100, 200, 300],
    index=["a", "b", "c"],
    name="amounts",
)
display(s_pd)
print(f"Index: {s_pd.index.tolist()}")
amounts
a100
b200
c300
Index: ['a', 'b', 'c']

Polars has no index concept. To replicate Pandas’ index behavior, pair the values with a label column in a DataFrame.

s_pl = pl.Series("amounts", [100, 200, 300])
display(s_pl)
df_pl = pl.DataFrame({"label": ["a", "b", "c"], "amounts": [100, 200, 300]})
display(df_pl)
amounts
i64
100
200
300
labelamounts
stri64
a100
b200
c300

Pandas / Polars | Data types — inference and casting

Pandas defaults to int64/float64/object; Polars defaults to Int64/Float64/String. Polars is stricter: no silent object fallback. When Pandas encounters mixed types in a column, it falls back to object dtype (a catch-all that can hold anything). Polars attempts to find a common supertype or raises an error.

Pandas object dtype is a catch-all

A column with object dtype can silently hold integers, strings, floats, and None in the same column. This defeats type checking and causes hard-to-debug issues downstream (e.g., 1 + "two" at runtime).

Use explicit dtypes or StringDtype

For text columns, use dtype="string" (or pd.StringDtype()) instead of the default object. For fully type-safe nullable types across all columns, use dtype_backend="pyarrow" when reading data.

mixed_pd = pd.Series([1, "two", 3.0], name="mixed")
print(f"dtype: {mixed_pd.dtype}")  # object  ← watch out!
display(mixed_pd)
dtype: object
mixed
01
1two
23.0

Polars attempts to find a common supertype for mixed-type lists. With strict=False, it coerces all values to string. With strict=True (default), it raises an error if no common numeric supertype exists.

try:
    mixed_pl = pl.Series("mixed", [1, "two", 3.0], strict=False)
    display(mixed_pl)
except Exception as e:
    print(f"Polars error: {e}")
mixed
str
1
two
3.0

Pandas uses .astype() for type casting and pd.array() with nullable dtypes for null-safe integer columns. Polars uses .cast() and has first-class null support — no special nullable dtype is needed.

s_pd = pd.Series([1, 2, 3], dtype="float32")
print(f"dtype after cast: {s_pd.dtype}")
 
# Convert to nullable Int
s_pd_nullable = pd.array([1, 2, None], dtype=pd.Int64Dtype())
print(f"Nullable Int64: {s_pd_nullable}")
dtype after cast: float32
Nullable Int64: <IntegerArray>
[1, 2, <NA>]
Length: 3, dtype: Int64
s_pl = pl.Series("vals", [1, 2, 3]).cast(pl.Float32)
print(f"dtype after cast: {s_pl.dtype}")
 
s_pl_null = pl.Series("vals", [1, 2, None])
print(f"dtype with null: {s_pl_null.dtype}")  # Int64, null is native
dtype after cast: Float32
dtype with null: Int64

Pandas / Polars | Basic Series operations

Both libraries provide aggregation methods (.sum(), .mean(), .std()) and .describe() for summary statistics. Polars’ .describe() additionally includes null_count, which Pandas omits.

prices_pd = pd.Series([10.5, 20.3, 30.1, 40.8, 50.0], name="price")
 
print("len   :", len(prices_pd))
print("shape :", prices_pd.shape)
print("sum   :", prices_pd.sum())
print("mean  :", prices_pd.mean())
print("std   :", prices_pd.std())
print("nunique:", prices_pd.nunique())
display(prices_pd.describe())
len   : 5
shape : (5,)
sum   : 151.7
mean  : 30.339999999999996
std   : 15.735405936930892
nunique: 5
price
count5.000000
mean30.340000
std15.735406
min10.500000
25%20.300000
50%30.100000
75%40.800000
max50.000000
prices_pl = pl.Series("price", [10.5, 20.3, 30.1, 40.8, 50.0])
 
print("len   :", prices_pl.len())
print("shape :", prices_pl.shape)
print("sum   :", prices_pl.sum())
print("mean  :", prices_pl.mean())
print("std   :", prices_pl.std())
print("nunique:", prices_pl.n_unique())
display(prices_pl.describe())
len   : 5
shape : (5,)
sum   : 151.7
mean  : 30.339999999999996
std   : 15.735405936930892
nunique: 5
statisticvalue
strf64
count5.0
null_count0.0
mean30.34
std15.735406
min10.5
25%20.3
50%30.1
75%40.8
max50.0

Pandas / Polars | Gotcha — NaN vs null

Pandas NaN silently promotes integers to float

Inserting a missing value into an integer Series causes Pandas to upcast the entire column to float64. This is because NaN is a float value in IEEE 754 — there is no integer NaN. This silent promotion can break join keys (1.0 != 1 in string comparisons) and accumulate floating-point error.

Use nullable integer dtypes or Polars

Pandas 2.x supports nullable integer types (pd.Int64Dtype()) that handle pd.NA without float promotion. Polars uses native Arrow null bitmasks — the dtype is always preserved.

s = pd.Series([1, 2, 3])
print(f"Before: {s.dtype}")  # int64
s.iloc[1] = np.nan # type: ignore
print(f"After:  {s.dtype}")  # float64  ← surprise!
display(s)
Before: int64
After:  float64
0
01.0
1NaN
23.0

Polars preserves the dtype — null is native and does not affect the column type. Use .scatter(index, None) to set a specific position to null.

s = pl.Series("x", [1, 2, 3])
s = s.scatter(1, None)
print(f"dtype: {s.dtype}")  # Int64  ← no promotion
display(s)
dtype: Int64
x
i64
1
null
3

DataFrame

A DataFrame is a two-dimensional table of columns. Pandas DataFrames have a row index for label-based alignment; Polars DataFrames do not — all data lives in columns.

Pandas vs Polars | Mutability

Pandas DataFrames are mutable — in-place operations like df["col"] = values modify the original object. Polars DataFrames are immutable — operations like .with_columns(), .filter(), and .sort() always return a new DataFrame. The original is never modified.

Pandas / Polars | Creating a DataFrame from a dict

data = {
    "symbol": ["AAPL", "MSFT", "GOOG", "AMZN"],
    "price":  [175.0, 340.0, 140.0, 180.0],
    "volume": [50_000_000, 30_000_000, 25_000_000, 40_000_000],
}
 
df_pd = pd.DataFrame(data)
print(f"type : {type(df_pd)}")
print(f"shape: {df_pd.shape}")
display(df_pd)
type : <class 'pandas.core.frame.DataFrame'>
shape: (4, 3)
symbolpricevolume
0AAPL175.050000000
1MSFT340.030000000
2GOOG140.025000000
3AMZN180.040000000
df_pl = pl.DataFrame(data)
print(f"type  : {type(df_pl)}")
print(f"shape : {df_pl.shape}")
print(f"height: {df_pl.height}, width: {df_pl.width}")
display(df_pl)
type  : <class 'polars.dataframe.frame.DataFrame'>
shape : (4, 3)
height: 4, width: 3
symbolpricevolume
strf64i64
AAPL175.050000000
MSFT340.030000000
GOOG140.025000000
AMZN180.040000000

Pandas / Polars | Creating a DataFrame from a list of dicts

records = [
    {"name": "Alice", "age": 30, "city": "London"},
    {"name": "Bob",   "age": 25, "city": "Paris"},
    {"name": "Carol", "age": 35, "city": "Berlin"},
]
 
df_pd = pd.DataFrame(records)
display(df_pd)
nameagecity
0Alice30London
1Bob25Paris
2Carol35Berlin
df_pl = pl.DataFrame(records)
display(df_pl)
nameagecity
stri64str
Alice30London
Bob25Paris
Carol35Berlin

Pandas / Polars | Creating a DataFrame from a NumPy array

arr = np.random.default_rng(42).standard_normal((5, 3))
 
df_pd = pd.DataFrame(arr, columns=["A", "B", "C"])
display(df_pd)
print(f"dtypes:\n{df_pd.dtypes}")
ABC
00.304717-1.0399840.750451
10.940565-1.951035-1.302180
20.127840-0.316243-0.016801
3-0.8530440.8793980.777792
40.0660311.1272410.467509
dtypes:
A    float64
B    float64
C    float64
dtype: object

Polars does not accept a raw NumPy 2D array directly — pass a dict mapping column names to array slices.

df_pl = pl.DataFrame({"A": arr[:, 0], "B": arr[:, 1], "C": arr[:, 2]})
display(df_pl)
print(f"dtypes: {df_pl.dtypes}")
ABC
f64f64f64
0.304717-1.0399840.750451
0.940565-1.951035-1.30218
0.12784-0.316243-0.016801
-0.8530440.8793980.777792
0.0660311.1272410.467509
dtypes: [Float64, Float64, Float64]

Pandas / Polars | Shape, height, width, column names

Pandas uses .shape and .columns. Polars adds .height and .width as explicit properties, plus .schema which returns a dict-like mapping of column names to types.

df_pd = pd.DataFrame(data)  # reuse earlier dict
 
print(f"shape   : {df_pd.shape}")
print(f"rows    : {df_pd.shape[0]}")
print(f"cols    : {df_pd.shape[1]}")
print(f"columns : {df_pd.columns.tolist()}")
print(f"dtypes  :\n{df_pd.dtypes}")
shape   : (4, 3)
rows    : 4
cols    : 3
columns : ['symbol', 'price', 'volume']
dtypes  :
symbol     object
price     float64
volume      int64
dtype: object
df_pl = pl.DataFrame(data)
 
print(f"shape   : {df_pl.shape}")
print(f"height  : {df_pl.height}")
print(f"width   : {df_pl.width}")
print(f"columns : {df_pl.columns}")
print(f"dtypes  : {df_pl.dtypes}")
print(f"schema  : {df_pl.schema}")
shape   : (4, 3)
height  : 4
width   : 3
columns : ['symbol', 'price', 'volume']
dtypes  : [String, Float64, Int64]
schema  : Schema({'symbol': String, 'price': Float64, 'volume': Int64})

The Index Concept

Pandas relies heavily on Index objects for alignment, selection, and joins. Polars has no index — all operations are column-based. This is one of the most significant design differences between the two libraries.

When does the index matter?

If your workflow involves time-series alignment (e.g., joining price data from different sources by date), Pandas’ automatic index alignment can be convenient. If you prefer explicit control over joins and merges, Polars’ index-free design avoids surprises from silent NaN injection.

Pandas Index basics

df = pd.DataFrame(
    {"value": [10, 20, 30]},
    index=pd.Index(["a", "b", "c"], name="key"),
)
display(df)
print(f"Index type : {type(df.index)}")
print(f"Index name : {df.index.name}")
print(f"Index vals : {df.index.tolist()}")
value
key
a10
b20
c30
Index type : <class 'pandas.core.indexes.base.Index'>
Index name : key
Index vals : ['a', 'b', 'c']

.set_index() promotes a column to the row index; .reset_index() moves the index back to a column. This is a common pattern when switching between label-based and positional access.

df_pd = pd.DataFrame({"key": ["a", "b", "c"], "value": [10, 20, 30]})
df_indexed = df_pd.set_index("key")
display(df_indexed)
 
df_reset = df_indexed.reset_index()
display(df_reset)
value
key
a10
b20
c30
keyvalue
0a10
1b20
2c30

Polars | No index, use columns instead

Polars keeps all data as regular columns. Filtering by a column value replaces Pandas’ .loc[] on an index. The foundation of all Polars operations is the expression API: pl.col("name") references a column by name and is the starting point for all transformations.

df_pl = pl.DataFrame({"key": ["a", "b", "c"], "value": [10, 20, 30]})
display(df_pl)
display(df_pl.filter(pl.col("key") == "b"))
keyvalue
stri64
a10
b20
c30
keyvalue
stri64
b20

Pandas | Gotcha — index alignment

Silent NaN injection from index alignment

When you combine two Pandas objects with different indexes, Pandas automatically aligns them by index label. Keys present in one but not the other produce NaN — silently expanding the result and potentially corrupting downstream computations.

Use explicit joins instead

For predictable behavior, use pd.merge() or .join() with explicit how= parameters instead of relying on automatic alignment. Polars avoids this entirely — addition is positional, and joins must be explicit.

s1 = pd.Series([1, 2, 3], index=["a", "b", "c"])
s2 = pd.Series([10, 20, 30], index=["b", "c", "d"])
 
result = s1 + s2
print("Index alignment produces NaN where keys don't overlap:")
display(result)
Index alignment produces NaN where keys don't overlap:
0
aNaN
b12.0
c23.0
dNaN

Polars has no alignment surprises — addition is purely positional. Both Series must have the same length.

s1 = pl.Series("s1", [1, 2, 3])
s2 = pl.Series("s2", [10, 20, 30])
result = s1 + s2
print("Polars addition is purely positional:")
display(result)
Polars addition is purely positional:
s1
i64
11
22
33

Pandas / Polars | MultiIndex vs grouped columns

Functional parity note

Pandas MultiIndex has no equivalent in the C# counterpart’s Deedle library. This is a Pandas-specific feature. Polars replaces MultiIndex with regular columns and group-by operations.

arrays = [
    ["bar", "bar", "baz", "baz"],
    ["one", "two", "one", "two"],
]
idx = pd.MultiIndex.from_arrays(arrays, names=["first", "second"])
df_mi = pd.DataFrame({"val": [10, 20, 30, 40]}, index=idx)
display(df_mi)
print(f"Index levels: {df_mi.index.nlevels}")
val
firstsecond
barone10
two20
bazone30
two40
Index levels: 2

In Polars, the equivalent is regular columns. Group-by operations replace MultiIndex workflows.

df_pl = pl.DataFrame({
    "first":  ["bar", "bar", "baz", "baz"],
    "second": ["one", "two", "one", "two"],
    "val":    [10, 20, 30, 40],
})
display(df_pl)
firstsecondval
strstri64
barone10
bartwo20
bazone30
baztwo40

Data Types Deep Dive

Understanding types is critical for data pipeline correctness. Pandas inherited NumPy types (int64, float64, object) plus its own Extension types (Int64, Float64, string, category). Polars uses Apache Arrow types exclusively (Int64, Float64, String, Date, Datetime, etc.), providing more precise control and consistent behavior.

Pandas / Polars | Listing available types

print("Pandas common dtypes:")
for dt in ["int64", "float64", "bool", "object", "datetime64[ns]",
           "timedelta64[ns]", "category", "string", "Int64", "Float64"]:
    print(f"  {dt}")
Pandas common dtypes:
  int64, float64, bool, object, datetime64[ns],
  timedelta64[ns], category, string, Int64, Float64
print("Polars common dtypes:")
for dt in [pl.Int8, pl.Int16, pl.Int32, pl.Int64,
           pl.UInt8, pl.UInt16, pl.UInt32, pl.UInt64,
           pl.Float32, pl.Float64,
           pl.Boolean, pl.String, pl.Date, pl.Datetime,
           pl.Duration, pl.Categorical, pl.Null]:
    print(f"  {dt}")
Polars common dtypes:
  Int8, Int16, Int32, Int64, UInt8, UInt16, UInt32, UInt64,
  Float32, Float64, Boolean, String, Date, Datetime,
  Duration, Categorical, Null

Pandas / Polars | Inspecting types on real data

Load a small reference dataset and compare how each library infers and reports column types.

df_pd = pd.read_csv(DATA / "dim_country.csv")
print(f"Shape: {df_pd.shape}")
display(df_pd.head())
display(df_pd.dtypes)

Shape: (212, 2)

country_nameiso_alpha2
0AfghanistanAF
1AlbaniaAL
2AlgeriaDZ
3American SamoaAS
4AndorraAD
0
country_nameobject
iso_alpha2object
df_pl = pl.read_csv(DATA / "dim_country.csv")
print(f"Shape: {df_pl.shape}")
display(df_pl.head())
print(f"Schema: {df_pl.schema}")

Shape: (212, 2)

country_nameiso_alpha2
strstr
AfghanistanAF
AlbaniaAL
AlgeriaDZ
American SamoaAS
AndorraAD

Schema: Schema({‘country_name’: String, ‘iso_alpha2’: String})

Pandas / Polars | Type casting

Pandas uses .astype() for column-level type conversion and pd.to_datetime() for date parsing. Polars uses .cast() within a .with_columns() expression, and .str.to_date() for date string parsing.

df_pd = pd.read_csv(DATA / "eurostoxx50_ohlcv.csv", nrows=5)
display(df_pd.dtypes)
 
df_pd_c = df_pd
df_pd_c["volume"] = df_pd_c["volume"].astype("float64")
df_pd_c["date"] = pd.to_datetime(df_pd_c["date"])
display(df_pd.dtypes)
0
idint64
symbolobject
dateobject
openfloat64
highfloat64
lowfloat64
closefloat64
adj_closefloat64
volumeint64
dividendsfloat64
stock_splitsfloat64
is_filledbool
0
idint64
symbolobject
datedatetime64[ns]
openfloat64
highfloat64
lowfloat64
closefloat64
adj_closefloat64
volumefloat64
dividendsfloat64
stock_splitsfloat64
is_filledbool
df_pl = pl.read_csv(DATA / "eurostoxx50_ohlcv.csv", n_rows=5)
print(df_pl.dtypes)
 
# Cast volume to Float64, date to Date
df_pl = df_pl.with_columns(
    pl.col("volume").cast(pl.Float64),
    pl.col("date").str.to_date("%Y-%m-%d"),
)
print(df_pl.dtypes)
display(df_pl)

[Int64, String, String, Float64, Float64, Float64, Float64, Float64, Int64, Float64, Float64, Boolean] [Int64, String, Date, Float64, Float64, Float64, Float64, Float64, Float64, Float64, Float64, Boolean]

idsymboldateopenhighlowcloseadj_closevolumedividendsstock_splitsis_filled
i64strdatef64f64f64f64f64f64f64f64bool
21160ABI.BR2021-01-0458.1558.8556.7857.2153.57611.513937e60.00.0false
21161ABI.BR2021-01-0556.957.9856.7557.1853.5481.382722e60.00.0false
21162ABI.BR2021-01-0657.9658.9457.3958.7755.0371.370204e60.00.0false
21163ABI.BR2021-01-0758.6858.8657.8858.454.69051.469911e60.00.0false
21164ABI.BR2021-01-0858.1658.457.4357.8654.18481.428681e60.00.0false

Pandas | Gotcha — object vs string dtype

Pandas object dtype accepts any Python type

Pandas’ default for text columns is object, which can silently hold integers, floats, None, and strings in the same column. Inserting a non-string value into an object column produces no error.

Use string dtype for type safety

Pass dtype="string" when creating the Series, or convert existing columns with .astype("string"). The StringDtype rejects non-string insertions.

s_obj = pd.Series(["a", "b", "c"])
s_str = pd.Series(["a", "b", "c"], dtype="string")
 
print(f"Default dtype: {s_obj.dtype}")   # object
print(f"String dtype:  {s_str.dtype}")   # string
 
# object allows mixed types — dangerous!
s_obj.iloc[0] = 42  # type: ignore # no error
print(f"After inserting int into object Series: {s_obj.tolist()}")
Default dtype: object
String dtype:  string
After inserting int into object Series: [42, 'b', 'c']

Loading Real Data from Multiple Formats

The ../data/ directory contains CSV, JSON, and Parquet files. Both Pandas and Polars can read all three formats, but with different APIs and performance characteristics.

Pandas / Polars | CSV

Both libraries read CSV through pd.read_csv() and pl.read_csv(). Polars is typically 3-10x faster on larger files due to multi-threaded parsing and zero-copy Arrow construction.

df_pd = pd.read_csv(DATA / "index_performance.csv")
print(f"Shape: {df_pd.shape}")
display(df_pd.head(3))

Shape: (5281, 15)

id_indexperf_datedaily_returncumulative_factorrolling_30d_returnrolling_90d_returnytd_returnrolling_30d_volatilitystocks_countavg_peavg_pbavg_dividend_yieldavg_market_cap_computed_at
01euro_stoxx_502021-01-05-0.0046260.995374NaNNaN-0.004626NaN49NaNNaNNaNNaN2026-03-04 22:40:26.069309
12euro_stoxx_502021-01-060.0183941.013683NaNNaN0.013683NaN48NaNNaNNaNNaN2026-03-04 22:40:26.069309
23euro_stoxx_502021-01-070.0054121.019168NaNNaN0.019168NaN49NaNNaNNaNNaN2026-03-04 22:40:26.069309
df_pl = pl.read_csv(DATA / "index_performance.csv")
print(f"Shape: {df_pl.shape}")
display(df_pl.head(3))

Shape: (5281, 15)

id_indexperf_datedaily_returncumulative_factorrolling_30d_returnrolling_90d_returnytd_returnrolling_30d_volatilitystocks_countavg_peavg_pbavg_dividend_yieldavg_market_cap_computed_at
i64strstrf64f64f64f64f64f64i64strstrstrstrstr
1euro_stoxx_502021-01-05-0.0046260.995374nullnull-0.004626null49nullnullnullnull2026-03-04 22:40:26.069309
2euro_stoxx_502021-01-060.0183941.013683nullnull0.013683null48nullnullnullnull2026-03-04 22:40:26.069309
3euro_stoxx_502021-01-070.0054121.019168nullnull0.019168null49nullnullnullnull2026-03-04 22:40:26.069309

Pandas / Polars | Parquet

Parquet preserves exact types, supports column projection, and is typically the fastest format to read for analytical workloads. Both pd.read_parquet() and pl.read_parquet() use the Apache Arrow Parquet reader under the hood.

df_pd = pd.read_parquet(DATA / "index_performance.parquet")
print(f"Shape: {df_pd.shape}")
display(df_pd.head(3))

Shape: (5281, 15)

id_indexperf_datedaily_returncumulative_factorrolling_30d_returnrolling_90d_returnytd_returnrolling_30d_volatilitystocks_countavg_peavg_pbavg_dividend_yieldavg_market_cap_computed_at
01euro_stoxx_502021-01-05-0.0046260.995374NaNNaN-0.004626NaN49NaNNaNNaNNaN2026-03-04 22:40:26.069309
12euro_stoxx_502021-01-060.0183941.013683NaNNaN0.013683NaN48NaNNaNNaNNaN2026-03-04 22:40:26.069309
23euro_stoxx_502021-01-070.0054121.019168NaNNaN0.019168NaN49NaNNaNNaNNaN2026-03-04 22:40:26.069309
df_pl = pl.read_parquet(DATA / "index_performance.parquet")
print(f"Shape: {df_pl.shape}")
display(df_pl.head(3))

Shape: (5281, 15)

id_indexperf_datedaily_returncumulative_factorrolling_30d_returnrolling_90d_returnytd_returnrolling_30d_volatilitystocks_countavg_peavg_pbavg_dividend_yieldavg_market_cap_computed_at
i64strdatef64f64f64f64f64f64i64f64f64f64f64datetime[ns]
1euro_stoxx_502021-01-05-0.0046260.995374nullnull-0.004626null49nullnullnullnull2026-03-04 22:40:26.069309
2euro_stoxx_502021-01-060.0183941.013683nullnull0.013683null48nullnullnullnull2026-03-04 22:40:26.069309
3euro_stoxx_502021-01-070.0054121.019168nullnull0.019168null49nullnullnullnull2026-03-04 22:40:26.069309

Pandas / Polars | JSON

Both libraries read JSON arrays of objects via pd.read_json() and pl.read_json(). Polars also supports lazy scanning of NDJSON (newline-delimited JSON) via pl.scan_ndjson().

df_pd = pd.read_json(DATA / "dim_country.json")
print(f"Shape: {df_pd.shape}")
display(df_pd.head(3))

Shape: (212, 2)

country_nameiso_alpha2
0AfghanistanAF
1AlbaniaAL
2AlgeriaDZ
df_pl = pl.read_json(DATA / "dim_country.json")
print(f"Shape: {df_pl.shape}")
display(df_pl.head(3))

Shape: (212, 2)

country_nameiso_alpha2
strstr
AfghanistanAF
AlbaniaAL
AlgeriaDZ

Inspecting DataFrames

After loading data, the first step is always inspection. Both libraries provide .head(), .tail(), .describe(), and memory usage estimation. Polars additionally provides .sample() with a seed parameter for reproducible random sampling.

Pandas / Polars | Head, tail, sample, describe

df_pd = pd.read_parquet(DATA / "eurostoxx50_ohlcv.parquet")
 
display(df_pd.head(3))
display(df_pd.tail(3))
display(df_pd.sample(3, random_state=42))
display(df_pd.describe())
idsymboldateopenhighlowcloseadj_closevolumedividendsstock_splitsis_filled
021160ABI.BR2021-01-0458.1558.8556.7857.2153.576115139370.00.0False
121161ABI.BR2021-01-0556.9057.9856.7557.1853.548013827220.00.0False
221162ABI.BR2021-01-0657.9658.9457.3958.7755.037013702040.00.0False
idsymboldateopenhighlowcloseadj_closevolumedividendsstock_splitsis_filled
6635266876WKL.AS2026-03-1068.869.1666.3467.1667.1613556450.00.0False
6635366877WKL.AS2026-03-1167.569.6067.0267.2267.2211425310.00.0False
6635466929WKL.AS2026-03-1267.067.5466.2867.3267.322103790.00.0False
idsymboldateopenhighlowcloseadj_closevolumedividendsstock_splitsis_filled
4305338920MUV2.DE2023-03-29320.0000322.600318.400322.4000290.32001958090.00.0False
562095772SAP.DE2022-11-0395.920096.34095.08095.510091.905214249730.00.0False
5351511015SAN.MC2022-09-152.59752.6862.5972.67652.3373701583490.00.0False
idopenhighlowcloseadj_closevolumedividendsstock_splits
count66355.00000066355.00000066355.00000066355.00000066355.00000066355.0000006.635500e+0466355.00000066355.000000
mean33179.733102197.040520199.364124194.585782197.034900190.4949095.942124e+060.0117570.000172
std19158.201385363.150484367.873829358.011643363.052047359.6353011.615619e+070.2831420.022716
min1.0000001.6010001.6628001.5842001.6066001.2013000.000000e+000.0000000.000000
25%16589.50000029.78995030.09000029.47000029.78745028.1434005.099855e+050.0000000.000000
50%33178.00000070.70000071.40000069.89000070.68000063.1410001.415896e+060.0000000.000000
75%49766.500000185.990000188.000000184.000000186.100000175.2539004.089299e+060.0000000.000000
max66930.0000002926.0000002957.0000002813.0000002839.0000002802.9382003.763915e+0822.5000005.000000
df_pl = pl.read_parquet(DATA / "eurostoxx50_ohlcv.parquet")
 
display(df_pl.head(3))
display(df_pl.tail(3))
display(df_pl.sample(3, seed=42))
display(df_pl.describe())
idsymboldateopenhighlowcloseadj_closevolumedividendsstock_splitsis_filled
i64strdatef64f64f64f64f64i64f64f64bool
21160ABI.BR2021-01-0458.1558.8556.7857.2153.576115139370.00.0false
21161ABI.BR2021-01-0556.957.9856.7557.1853.54813827220.00.0false
21162ABI.BR2021-01-0657.9658.9457.3958.7755.03713702040.00.0false
idsymboldateopenhighlowcloseadj_closevolumedividendsstock_splitsis_filled
i64strdatef64f64f64f64f64i64f64f64bool
66876WKL.AS2026-03-1068.869.1666.3467.1667.1613556450.00.0false
66877WKL.AS2026-03-1167.569.667.0267.2267.2211425310.00.0false
66929WKL.AS2026-03-1267.067.5466.2867.3267.322103790.00.0false
idsymboldateopenhighlowcloseadj_closevolumedividendsstock_splitsis_filled
i64strdatef64f64f64f64f64i64f64f64bool
11531SAN.MC2024-09-204.584.62854.55854.55854.3259709611830.00.0false
35605CS.PA2025-10-1540.5441.040.1740.1740.1732045820.00.0false
63668WKL.AS2022-01-0797.5297.9696.9297.3491.08084084110.00.0false
statisticidsymboldateopenhighlowcloseadj_closevolumedividendsstock_splitsis_filled
strf64strstrf64f64f64f64f64f64f64f64f64
count66355.0663556635566355.066355.066355.066355.066355.066355.066355.066355.066355.0
null_count0.0000.00.00.00.00.00.00.00.00.0
mean33179.733102null2023-08-05 00:56:42.354005197.04052199.364124194.585782197.0349190.4949095.9421e60.0117570.0001720.00009
std19158.201385nullnull363.150484367.873829358.011643363.052047359.6353011.6156e70.2831420.022716null
min1.0ABI.BR2021-01-041.6011.66281.58421.60661.20130.00.00.00.0
25%16590.0null2022-04-2029.7930.0929.4729.789928.1461509991.00.00.0null
50%33178.0null2023-08-0370.771.469.8970.6863.1411.415896e60.00.0null
75%49767.0null2024-11-19186.0188.0184.0186.1175.26094.089463e60.00.0null
max66930.0WKL.AS2026-03-122926.02957.02813.02839.02802.93823.76391539e822.55.01.0

Pandas / Polars | Memory usage

Pandas reports memory usage via .info(memory_usage="deep") which accounts for Python object overhead. Polars uses .estimated_size() which reports the raw Arrow buffer size — typically smaller because Arrow avoids per-element Python object overhead.

print("Pandas memory usage:")
df_pd.info(memory_usage="deep")

Pandas memory usage: <class ‘pandas.core.frame.DataFrame’> RangeIndex: 66355 entries, 0 to 66354 Data columns (total 12 columns):

Column Non-Null Count Dtype

 0   id            66355 non-null  int64
 1   symbol        66355 non-null  object
 2   date          66355 non-null  object
 3   open          66355 non-null  float64
 4   high          66355 non-null  float64
 5   low           66355 non-null  float64
 6   close         66355 non-null  float64
 7   adj_close     66355 non-null  float64
 8   volume        66355 non-null  int64
 9   dividends     66355 non-null  float64
 10  stock_splits  66355 non-null  float64
 11  is_filled     66355 non-null  bool
dtypes: bool(1), float64(7), int64(2), object(2)
memory usage: 10.6 MB

size_bytes = df_pl.estimated_size("b")
size_mb = df_pl.estimated_size("mb")
print(f"Polars estimated size: {size_bytes:,} bytes ({size_mb:.2f} MB)")

Polars estimated size: 5,454,618 bytes (5.20 MB)

Pandas / Polars | Null and NaN inspection

Check for null counts across all columns to assess data quality.

print("Null counts per column:")
display(df_pd.isnull().sum())
print(f"\nTotal nulls: {df_pd.isnull().sum().sum()}")

Null counts per column:

0
id0
symbol0
date0
open0
high0
low0
close0
adj_close0
volume0
dividends0
stock_splits0
is_filled0

Total nulls: 0

Polars provides .null_count() which returns a single-row DataFrame showing null counts per column.

print("Null counts per column:")
display(df_pl.null_count())

Null counts per column:

idsymboldateopenhighlowcloseadj_closevolumedividendsstock_splitsis_filled
u32u32u32u32u32u32u32u32u32u32u32u32
000000000000

Edge Cases and Gotchas

Common pitfalls when working with Pandas and Polars, and how the two libraries handle them differently.

Pandas / Polars | Empty DataFrames

Both libraries support creating empty DataFrames with a predefined schema — useful as sentinel values or accumulator start states.

df_empty_pd = pd.DataFrame({"a": pd.Series(dtype="int64"), "b": pd.Series(dtype="float64")})
print(f"Shape: {df_empty_pd.shape}")
display(df_empty_pd.dtypes)

Shape: (0, 2)

0
aint64
bfloat64
df_empty_pl = pl.DataFrame(schema={"a": pl.Int64, "b": pl.Float64})
print(f"Shape: {df_empty_pl.shape}")
print(f"Schema: {df_empty_pl.schema}")
display(df_empty_pl)

Shape: (0, 2) Schema: Schema({‘a’: Int64, ‘b’: Float64})

ab
i64f64

Pandas / Polars | Column name duplicates

Pandas allows duplicate column names

Creating a DataFrame with duplicate column names is silently accepted. Selecting by name then returns multiple columns instead of one — a common source of hard-to-debug errors.

Polars rejects duplicate column names at creation

Polars raises a SchemaError if you attempt to create a DataFrame with duplicate column names. This catches the bug immediately at construction time.

df_dup = pd.DataFrame([[1, 2]], columns=["x", "x"])
display(df_dup)
print(f"Selecting 'x' returns {df_dup['x'].shape[1]} columns — not 1!")
xx
012

Selecting ‘x’ returns 2 columns — not 1!

# Polars rejects duplicate column names
try:
    df_dup = pl.DataFrame({"x": [1], "x": [2]})  # dict deduplicates first
    print("Dict deduplicates, so only one 'x':")
    display(df_dup)
except Exception as e:
    print(f"Error: {e}")
 
# Trying via schema
try:
    df_dup = pl.from_records([(1, 2)], schema=["x", "x"], orient="row")
    display(df_dup)
except Exception as e:
    print(f"Polars error on duplicate columns: {e}")

Dict deduplicates, so only one ‘x’:

x
i64
2

Polars error on duplicate columns: column with name ‘x’ has more than one occurrence

Pandas / Polars | Integer overflow

Pandas silently wraps on integer overflow

Adding 1 to int64 max value wraps around to the most negative integer — no error, no warning. This is inherited from NumPy’s C-level integer arithmetic.

Polars detects overflow

In debug/development builds, Polars raises an error on integer overflow. In release builds, it may return the wrapped value but the behavior is documented and consistent.

s = pd.Series([np.iinfo(np.int64).max], dtype="int64")
print(f"Max int64: {s.iloc[0]}")
s_overflow = s + 1
print(f"Max + 1  : {s_overflow.iloc[0]}  ← silent wrap!")

Max int64: 9223372036854775807 Max + 1 : -9223372036854775808 ← silent wrap!

# Polars raises on overflow in debug builds / returns null
s = pl.Series("x", [2**63 - 1], dtype=pl.Int64)
print(f"Max int64: {s[0]}")
try:
    result = s + 1
    print(f"Max + 1  : {result[0]}")
except Exception as e:
    print(f"Polars overflow error: {e}")

Max int64: 9223372036854775807 Max + 1 : -9223372036854775808

Pandas / Polars | .values vs .to_numpy() vs .to_list()

Pandas .values may return a view — mutations propagate

.values returns a NumPy array that may share memory with the Series. Mutating the array silently mutates the original Series. Use .to_numpy() (recommended) or .to_list() for a safe copy.

Polars .to_numpy() always returns a copy

Polars Series are backed by Arrow arrays (immutable). .to_numpy() always copies the data — mutations to the array never affect the original Series.

s = pd.Series([1, 2, 3])
 
print(f".values type      : {type(s.values)}")
print(f".to_numpy() type  : {type(s.to_numpy())}")
print(f".to_list() type   : {type(s.to_list())}")
 
arr = s.values
arr[0] = 999
print(f"Series after mutating .values: {s.tolist()}  ← changed!")

.values type : <class ‘numpy.ndarray’> .to_numpy() type : <class ‘numpy.ndarray’> .to_list() type : <class ‘list’> Series after mutating .values: [999, 2, 3] ← changed!

s = pl.Series("x", [1, 2, 3])
 
# Explicitly create a writable copy in memory
arr = s.to_numpy().copy()
 
# Now mutation is allowed
arr[0] = 999
 
print(f"Original Series: {s.to_list()} -> unchanged")
print(f"Mutated Array: {arr.tolist()}")

Original Series: [1, 2, 3] → unchanged Mutated Array: [999, 2, 3]

Comparison Summary

comparison = pl.DataFrame({
    "Feature": [
        "1-D data structure",
        "2-D data structure",
        "Row index",
        "Missing values",
        "Default int type",
        "Default float type",
        "Default string type",
        "Type safety",
        "Duplicate column names",
        "Memory layout",
        "Lazy evaluation",
        "MultiIndex",
        "Create from dict",
        "Create from numpy",
        "Create from records",
        "Shape attribute",
        "Height / width attrs",
        "Null counting",
        "Memory estimation",
        "Type casting",
    ],
    "Pandas": [
        "pd.Series (indexed)",
        "pd.DataFrame (indexed)",
        "Yes — RangeIndex, named, Multi",
        "NaN (float) or pd.NA",
        "int64",
        "float64",
        "object (or StringDtype)",
        "Low — object dtype is a catch-all",
        "Allowed (bug-prone)",
        "Column-major (BlockManager)",
        "No (eager only)",
        "Yes — pd.MultiIndex",
        "pd.DataFrame(dict)",
        "pd.DataFrame(arr, columns=…)",
        "pd.DataFrame(list_of_dicts)",
        ".shape → (rows, cols)",
        "No",
        "df.isnull().sum()",
        "df.memory_usage(deep=True)",
        ".astype() / pd.to_datetime()",
    ],
    "Polars": [
        "pl.Series (named, no index)",
        "pl.DataFrame (no index)",
        "No — all data lives in columns",
        "null (Arrow bitmask)",
        "Int64",
        "Float64",
        "String (Utf8)",
        "High — strict type checking",
        "Rejected (error)",
        "Column-major (Arrow arrays)",
        "Yes — pl.LazyFrame",
        "No — use regular columns",
        "pl.DataFrame(dict)",
        "pl.DataFrame({'col': arr})",
        "pl.DataFrame(list_of_dicts)",
        ".shape → (rows, cols)",
        "Yes — .height, .width",
        "df.null_count()",
        "df.estimated_size()",
        ".cast() / .str.to_date()",
    ],
})
 
display(comparison)
FeaturePandasPolars
strstrstr
1-D data structurepd.Series (indexed)pl.Series (named, no index)
2-D data structurepd.DataFrame (indexed)pl.DataFrame (no index)
Row indexYes — RangeIndex, named, MultiNo — all data lives in columns
Missing valuesNaN (float) or pd.NAnull (Arrow bitmask)
Default int typeint64Int64
Default float typefloat64Float64
Default string typeobject (or StringDtype)String (Utf8)
Type safetyLow — object dtype is a catch-…High — strict type checking
Duplicate column namesAllowed (bug-prone)Rejected (error)
Memory layoutColumn-major (BlockManager)Column-major (Arrow arrays)
Lazy evaluationNo (eager only)Yes — pl.LazyFrame
MultiIndexYes — pd.MultiIndexNo — use regular columns
Create from dictpd.DataFrame(dict)pl.DataFrame(dict)
Create from numpypd.DataFrame(arr, columns=…)pl.DataFrame({'col': arr})
Create from recordspd.DataFrame(list_of_dicts)pl.DataFrame(list_of_dicts)
Shape attribute.shape → (rows, cols).shape → (rows, cols)
Height / width attrsNoYes — .height, .width
Null countingdf.isnull().sum()df.null_count()
Memory estimationdf.memory_usage(deep=True)df.estimated_size()
Type casting.astype() / pd.to_datetime().cast() / .str.to_date()

Key Takeaways

  • Polars has no index — this eliminates a whole class of alignment bugs.
  • Polars uses Arrow-native nulls — no NaN-induced type promotion.
  • Polars is stricter with types — catches errors earlier.
  • Pandas is more permissive — great for exploration, risky in production.
  • Both can create DataFrames from dicts, lists, numpy, and files.
  • For new projects, Polars’ design avoids many Pandas footguns while being faster.

Reading & Writing Data

This section covers I/O operations: discovering data files, reading from CSV/JSON/Parquet, writing output, lazy scanning, and format benchmarks.

Discovering Data Files

We use pathlib and glob patterns to discover every file in the ../data/ directory, grouped by extension.

DATA_DIR = Path("../data")
assert DATA_DIR.exists(), f"Data directory not found: {DATA_DIR.resolve()}"
 
all_files = sorted(DATA_DIR.iterdir())
print(f"Total files in data directory: {len(all_files)}")
for f in all_files:
    size_kb = f.stat().st_size / 1024
    print(f"  {f.name:<35s} {size_kb:>10,.1f} KB")

Total files in data directory: 39 dim_country.csv 2.8 KB dim_country.json 13.0 KB dim_country.parquet 5.0 KB dim_index.csv 0.2 KB dim_index.json 0.6 KB dim_index.parquet 3.5 KB eurostoxx50_ohlcv.csv 5,162.0 KB eurostoxx50_ohlcv.json 17,668.3 KB eurostoxx50_ohlcv.parquet 2,426.7 KB index_dim.csv 278.0 KB index_dim.json 371.8 KB index_dim.parquet 145.0 KB index_performance.csv 940.1 KB index_performance.json 2,432.6 KB index_performance.parquet 344.9 KB oil20_ohlcv.csv 1,849.1 KB oil20_ohlcv.json 6,511.6 KB oil20_ohlcv.parquet 882.4 KB pulse.csv 6.8 KB pulse.json 20.8 KB pulse.parquet 16.4 KB scores_daily.csv 236.6 KB scores_daily.json 541.0 KB scores_daily.parquet 117.4 KB scores_quarterly.csv 49.0 KB scores_quarterly.json 148.1 KB scores_quarterly.parquet 33.6 KB signals_daily.csv 84.6 KB signals_daily.json 270.5 KB signals_daily.parquet 59.4 KB signals_quarterly.csv 28.4 KB signals_quarterly.json 112.7 KB signals_quarterly.parquet 29.2 KB stoxxusa50_ohlcv.csv 5,048.3 KB stoxxusa50_ohlcv.json 17,318.1 KB stoxxusa50_ohlcv.parquet 2,522.3 KB trading_calendar.csv 1,498.8 KB trading_calendar.json 7,342.8 KB trading_calendar.parquet 34.8 KB

# Group files by extension using glob
csv_files  = sorted(DATA_DIR.glob("*.csv"))
json_files = sorted(DATA_DIR.glob("*.json"))
pq_files   = sorted(DATA_DIR.glob("*.parquet"))
 
print(f"CSV  files: {len(csv_files)}")
print(f"JSON files: {len(json_files)}")
print(f"Parquet files: {len(pq_files)}")

CSV files: 13 JSON files: 13 Parquet files: 13

# Build a summary table of file sizes by format
rows = []
stems = sorted({f.stem for f in all_files})
for stem in stems:
    row = {"dataset": stem}
    for ext in ["csv", "json", "parquet"]:
        p = DATA_DIR / f"{stem}.{ext}"
        row[ext + "_KB"] = round(p.stat().st_size / 1024, 1) if p.exists() else None # type: ignore
    rows.append(row)
 
size_df = pd.DataFrame(rows)
display(Markdown("### File sizes by format (KB)"))
display(size_df)

File sizes by format (KB)

datasetcsv_KBjson_KBparquet_KB
0dim_country2.813.05.0
1dim_index0.20.63.5
2eurostoxx50_ohlcv5162.017668.32426.7
3index_dim278.0371.8145.0
4index_performance940.12432.6344.9
5oil20_ohlcv1849.16511.6882.4
6pulse6.820.816.4
7scores_daily236.6541.0117.4
8scores_quarterly49.0148.133.6
9signals_daily84.6270.559.4
10signals_quarterly28.4112.729.2
11stoxxusa50_ohlcv5048.317318.12522.3
12trading_calendar1498.87342.834.8

Reading CSV Files

Pandas | read_csv

read_csv() dtype inference trap

pd.read_csv() infers dtypes from the first 100 rows by default. If the first 100 rows of a column contain only integers but row 101 has a float or null, Pandas silently coerces the entire column. Always specify dtype= for critical columns, or use dtype_backend="pyarrow" for consistent nullable types. Integer columns with any null values are silently upcast to float64 — a common source of broken join keys (1.0 != 1 in string comparisons).

Always declare dtypes for critical columns

Pass an explicit dtype= dict for columns used as join keys or numeric computations: pd.read_csv(path, dtype={"id": "int64", "isin": "str"}). For fully safe nullable types across all columns, use dtype_backend="pyarrow" — integer columns with nulls stay int64[pyarrow] instead of being silently upcast to float64.

Encoding defaults differ between Pandas

Encoding defaults differ between Pandas and Polars Pandas read_csv() defaults to encoding='utf-8' but silently falls back on some platforms. Polars only supports UTF-8 — non-UTF-8 files raise an error immediately. For files from legacy systems (SQL Server BCP exports, Excel CSV), always specify encoding='utf-8-sig' (to handle BOM) or encoding='latin-1'.

Specify encoding explicitly for legacy sources

Always pass encoding= when reading files from SQL Server BCP exports, Excel CSV, or any legacy system: pd.read_csv(path, encoding='utf-8-sig') handles BOM-prefixed UTF-8; use encoding='latin-1' for Western European legacy files. For Polars, pre-convert non-UTF-8 files with iconv or Python’s codecs module before ingestion.

Reads dim_country.csv with default inference (212 rows × 2 columns), then reads eurostoxx50_ohlcv.csv with dtype={"ticker": "category"}, parse_dates=["date"], and na_values to override defaults — demonstrating how explicit parameters prevent silent dtype coercion and missed null sentinels at load time.

df_pd = pd.read_csv(DATA_DIR / "dim_country.csv")
print(f"Shape: {df_pd.shape}")
print(f"Dtypes:\n{df_pd.dtypes}")
display(df_pd.head())

Shape: (212, 2) Dtypes: country_name object iso_alpha2 object dtype: object

country_nameiso_alpha2
0AfghanistanAF
1AlbaniaAL
2AlgeriaDZ
3American SamoaAS
4AndorraAD

Read a larger file with explicit parameters: dtype for categorical columns, parse_dates for date detection, and na_values to specify additional null sentinels.

df_pd_ohlcv = pd.read_csv(
    DATA_DIR / "eurostoxx50_ohlcv.csv",
    sep=",",               # separator (default)
    dtype={"ticker": "category"},
    parse_dates=["date"],
    na_values=["", "NA", "N/A"],
)
print(f"Shape: {df_pd_ohlcv.shape}")
print(f"Dtypes:\n{df_pd_ohlcv.dtypes}")
display(df_pd_ohlcv.head(3))

Shape: (66355, 12) Dtypes: id int64 symbol object date datetime64[ns] open float64 high float64 low float64 close float64 adj_close float64 volume int64 dividends float64 stock_splits float64 is_filled bool dtype: object

idsymboldateopenhighlowcloseadj_closevolumedividendsstock_splitsis_filled
021160ABI.BR2021-01-0458.1558.8556.7857.2153.576115139370.00.0False
121161ABI.BR2021-01-0556.9057.9856.7557.1853.548013827220.00.0False
221162ABI.BR2021-01-0657.9658.9457.3958.7755.037013702040.00.0False

Use usecols to read only specific columns and nrows to limit rows — useful for peeking at large files without loading everything.

df_peek = pd.read_csv(
    DATA_DIR / "trading_calendar.csv",
    usecols=lambda c: c in ["date", "exchange", "is_open"],
    nrows=5,
)
display(Markdown("### Peek at trading_calendar.csv (first 5 rows, selected cols)"))
display(df_peek)

Peek at trading_calendar.csv (first 5 rows, selected cols)

date
02021-01-01
12021-01-02
22021-01-03
32021-01-04
42021-01-05

Polars | read_csv (Eager)

Polars’ eager read_csv() reads the entire file into memory. It uses multi-threaded parsing and infers types from the first 1000 rows by default. Use try_parse_dates=True for automatic date detection.

Reads dim_country.csv to confirm a 2-column String schema, then reads eurostoxx50_ohlcv.csv with try_parse_dates=True and schema_overrides={"ticker": pl.Categorical} — producing a 66 355-row DataFrame where date is automatically typed as Date and ticker as Categorical.

df_pl = pl.read_csv(DATA_DIR / "dim_country.csv")
print(f"Shape: {df_pl.shape}")
print(f"Schema: {df_pl.schema}")
display(df_pl.head())

Shape: (212, 2) Schema: Schema({‘country_name’: String, ‘iso_alpha2’: String})

country_nameiso_alpha2
strstr
AfghanistanAF
AlbaniaAL
AlgeriaDZ
American SamoaAS
AndorraAD

Use try_parse_dates, null_values, and schema_overrides for more controlled parsing.

df_pl_ohlcv = pl.read_csv(
    DATA_DIR / "eurostoxx50_ohlcv.csv",
    separator=",",
    null_values=["", "NA", "N/A"],
    try_parse_dates=True,
    schema_overrides={"ticker": pl.Categorical},
)
print(f"Shape: {df_pl_ohlcv.shape}")
print(f"Schema: {df_pl_ohlcv.schema}")
display(df_pl_ohlcv.head(3))

Shape: (66355, 12) Schema: Schema({‘id’: Int64, ‘symbol’: String, ‘date’: Date, ‘open’: Float64, ‘high’: Float64, ‘low’: Float64, ‘close’: Float64, ‘adj_close’: Float64, ‘volume’: Int64, ‘dividends’: Float64, ‘stock_splits’: Float64, ‘is_filled’: Boolean})

idsymboldateopenhighlowcloseadj_closevolumedividendsstock_splitsis_filled
i64strdatef64f64f64f64f64i64f64f64bool
21160ABI.BR2021-01-0458.1558.8556.7857.2153.576115139370.00.0false
21161ABI.BR2021-01-0556.957.9856.7557.1853.54813827220.00.0false
21162ABI.BR2021-01-0657.9658.9457.3958.7755.03713702040.00.0false

Polars | scan_csv (Lazy)

scan_csv() returns a LazyFrame — no data is read until .collect() is called. The query optimizer can push predicates and projections down to the scan, reading only what’s needed.

Creates a LazyFrame from eurostoxx50_ohlcv.csv without reading any row data, then filters to ADYEN.AS, selects 4 columns, and collects — confirming the 12-column schema is available immediately while data is only read at .collect() time.

lf = pl.scan_csv(DATA_DIR / "eurostoxx50_ohlcv.csv", try_parse_dates=True)
print(f"Type: {type(lf)}")
print(f"Schema: {lf.collect_schema()}")
print("No data loaded yet - this is a query plan.")

Type: <class ‘polars.lazyframe.frame.LazyFrame’> Schema: Schema({‘id’: Int64, ‘symbol’: String, ‘date’: Date, ‘open’: Float64, ‘high’: Float64, ‘low’: Float64, ‘close’: Float64, ‘adj_close’: Float64, ‘volume’: Int64, ‘dividends’: Float64, ‘stock_splits’: Float64, ‘is_filled’: Boolean}) No data loaded yet - this is a query plan.

Collect a filtered subset — Polars pushes the predicate down to the file scan.

result = (
    lf
    .filter(pl.col("symbol") == "ADYEN.AS")
    .select("date", "symbol", "close", "volume")
    .head(5)
    .collect()
)
display(Markdown("### Lazy scan -> filtered collect"))
display(result)

Lazy scan → filtered collect

datesymbolclosevolume
datestrf64i64
2021-01-04ADYEN.AS1859.599408
2021-01-05ADYEN.AS1829.086256
2021-01-06ADYEN.AS1733.0156844
2021-01-07ADYEN.AS1714.590183
2021-01-08ADYEN.AS1756.597176
# Load ALL csv files with Pandas
print("Loading all CSV files with Pandas...")
pd_csvs = {}
for f in csv_files:
    t0 = time.perf_counter()
    pd_csvs[f.stem] = pd.read_csv(f)
    elapsed = time.perf_counter() - t0
    print(f"  {f.stem:30s} -> {pd_csvs[f.stem].shape}  ({elapsed:.3f}s)")

Loading all CSV files with Pandas… dim_country → (212, 2) (0.001s) dim_index → (4, 5) (0.001s) eurostoxx50_ohlcv → (66355, 12) (0.035s) index_dim → (169, 26) (0.004s) index_performance → (5281, 15) (0.006s) oil20_ohlcv → (24738, 12) (0.013s) pulse → (40, 20) (0.001s) scores_daily → (466, 36) (0.003s) scores_quarterly → (170, 29) (0.001s) signals_daily → (466, 19) (0.001s) signals_quarterly → (177, 22) (0.001s) stoxxusa50_ohlcv → (65100, 12) (0.029s) trading_calendar → (29335, 11) (0.010s)

# Load ALL csv files with Polars
print("Loading all CSV files with Polars...")
pl_csvs = {}
for f in csv_files:
    t0 = time.perf_counter()
    pl_csvs[f.stem] = pl.read_csv(f, try_parse_dates=True)
    elapsed = time.perf_counter() - t0
    print(f"  {f.stem:30s} -> {pl_csvs[f.stem].shape}  ({elapsed:.3f}s)")

Loading all CSV files with Polars… dim_country → (212, 2) (0.001s) dim_index → (4, 5) (0.001s) eurostoxx50_ohlcv → (66355, 12) (0.003s) index_dim → (169, 26) (0.009s) index_performance → (5281, 15) (0.003s) oil20_ohlcv → (24738, 12) (0.002s) pulse → (40, 20) (0.001s) scores_daily → (466, 36) (0.003s) scores_quarterly → (170, 29) (0.002s) signals_daily → (466, 19) (0.001s) signals_quarterly → (177, 22) (0.001s) stoxxusa50_ohlcv → (65100, 12) (0.003s) trading_calendar → (29335, 11) (0.002s)

Reading JSON Files

Pandas | read_json

Reads dim_index.json (4 rows × 5 columns) and index_performance.json (5 281 rows × 15 columns), printing shapes and dtypes — showing that Pandas infers date-like strings as object and mixes int64, float64, and datetime64[ns] without an explicit schema.

df_pd_json = pd.read_json(DATA_DIR / "dim_index.json")
print(f"Shape: {df_pd_json.shape}")
display(df_pd_json.head())

Shape: (4, 5)

index_keydisplay_namefile_prefixcolorcurrency
0euro_stoxx_50Euro Stoxx 50eurostoxx50#4285F4€
1oil_20Oil & Gas 20oil20#D4A017$
2stoxx_asia_50STOXX Asia/Pacific 50stoxxasia50#EF5350
3stoxx_usa_50STOXX USA 50stoxxusa50#FFFFFF$
# Load a larger JSON file
df_pd_perf = pd.read_json(DATA_DIR / "index_performance.json")
print(f"Shape: {df_pd_perf.shape}")
print(f"Dtypes:\n{df_pd_perf.dtypes}")
display(df_pd_perf.head(3))

Shape: (5281, 15) Dtypes: id int64 _index object perf_date object daily_return float64 cumulative_factor float64 rolling_30d_return float64 rolling_90d_return float64 ytd_return float64 rolling_30d_volatility float64 stocks_count int64 avg_pe float64 avg_pb float64 avg_dividend_yield float64 avg_market_cap float64 _computed_at datetime64[ns] dtype: object

id_indexperf_datedaily_returncumulative_factorrolling_30d_returnrolling_90d_returnytd_returnrolling_30d_volatilitystocks_countavg_peavg_pbavg_dividend_yieldavg_market_cap_computed_at
01euro_stoxx_502021-01-05-0.0046260.995374NaNNaN-0.004626NaN49NaNNaNNaNNaN2026-03-04 22:40:26.069309
12euro_stoxx_502021-01-060.0183941.013683NaNNaN0.013683NaN48NaNNaNNaNNaN2026-03-04 22:40:26.069309
23euro_stoxx_502021-01-070.0054121.019168NaNNaN0.019168NaN49NaNNaNNaNNaN2026-03-04 22:40:26.069309

Polars | read_json

Reads dim_index.json with default inference (4 rows × 5-column String schema), then reads index_performance.json with infer_schema_length=None to force full-file scanning — preventing ComputeError from early-null columns in the 5 281-row performance dataset.

df_pl_json = pl.read_json(DATA_DIR / "dim_index.json")
print(f"Shape: {df_pl_json.shape}")
print(f"Schema: {df_pl_json.schema}")
display(df_pl_json.head())

Shape: (4, 5) Schema: Schema({‘index_key’: String, ‘display_name’: String, ‘file_prefix’: String, ‘color’: String, ‘currency’: String})

index_keydisplay_namefile_prefixcolorcurrency
strstrstrstrstr
euro_stoxx_50Euro Stoxx 50eurostoxx50#4285F4€
oil_20Oil & Gas 20oil20#D4A017$
stoxx_asia_50STOXX Asia/Pacific 50stoxxasia50#EF5350
stoxx_usa_50STOXX USA 50stoxxusa50#FFFFFF$
# Setting infer_schema_length to None forces Polars to scan the whole file
df_pl_perf = pl.read_json(
    DATA_DIR / "index_performance.json",
    infer_schema_length=None
)
 
print(f"Shape: {df_pl_perf.shape}")
display(df_pl_perf.head(3))

Shape: (5281, 15)

id_indexperf_datedaily_returncumulative_factorrolling_30d_returnrolling_90d_returnytd_returnrolling_30d_volatilitystocks_countavg_peavg_pbavg_dividend_yieldavg_market_cap_computed_at
i64strstrf64f64f64f64f64f64i64f64f64f64f64str
1euro_stoxx_502021-01-05-0.0046260.995374nullnull-0.004626null49nullnullnullnull2026-03-04 22:40:26.069309
2euro_stoxx_502021-01-060.0183941.013683nullnull0.013683null48nullnullnullnull2026-03-04 22:40:26.069309
3euro_stoxx_502021-01-070.0054121.019168nullnull0.019168null49nullnullnullnull2026-03-04 22:40:26.069309

Polars | scan_ndjson (Lazy)

scan_ndjson works with newline-delimited JSON files. Standard JSON arrays need to be converted first. We demonstrate by writing NDJSON and scanning it back.

Converts dim_index.json to NDJSON via .write_ndjson(), then scans the resulting file lazily with scan_ndjson and collects only 3 rows — confirming the 5-column String schema is available without loading the full file, then deletes the temp file.

# Write an NDJSON file from an existing dataframe, then scan it lazily
ndjson_path = DATA_DIR / "dim_index.ndjson"
pl.read_json(DATA_DIR / "dim_index.json").write_ndjson(ndjson_path)
 
lf_ndjson = pl.scan_ndjson(ndjson_path)
print(f"Type: {type(lf_ndjson)}")
print(f"Schema: {lf_ndjson.collect_schema()}")
display(lf_ndjson.head(3).collect())
 
# Clean up temp file
ndjson_path.unlink()

Type: <class ‘polars.lazyframe.frame.LazyFrame’> Schema: Schema({‘index_key’: String, ‘display_name’: String, ‘file_prefix’: String, ‘color’: String, ‘currency’: String})

index_keydisplay_namefile_prefixcolorcurrency
strstrstrstrstr
euro_stoxx_50Euro Stoxx 50eurostoxx50#4285F4€
oil_20Oil & Gas 20oil20#D4A017$
stoxx_asia_50STOXX Asia/Pacific 50stoxxasia50#EF5350
# Load ALL json files with Pandas
print("Loading all JSON files with Pandas...")
pd_jsons = {}
for f in json_files:
    t0 = time.perf_counter()
    pd_jsons[f.stem] = pd.read_json(f)
    elapsed = time.perf_counter() - t0
    print(f"  {f.stem:30s} -> {pd_jsons[f.stem].shape}  ({elapsed:.3f}s)")

Loading all JSON files with Pandas… dim_country → (212, 2) (0.002s) dim_index → (4, 5) (0.001s) eurostoxx50_ohlcv → (66355, 12) (0.173s) index_dim → (169, 26) (0.004s) index_performance → (5281, 15) (0.016s) oil20_ohlcv → (24738, 12) (0.057s) pulse → (40, 20) (0.003s) scores_daily → (466, 36) (0.008s) scores_quarterly → (170, 29) (0.004s) signals_daily → (466, 19) (0.003s) signals_quarterly → (177, 22) (0.003s) stoxxusa50_ohlcv → (65100, 12) (0.153s) trading_calendar → (29335, 11) (0.062s)

# Load ALL json files with Polars
print("Loading all JSON files with Polars...")
pl_jsons = {}
 
for f in json_files:
    t0 = time.perf_counter()
 
    # Force full-file schema scanning to prevent the Null to Float ComputeError
    pl_jsons[f.stem] = pl.read_json(f, infer_schema_length=None)
 
    elapsed = time.perf_counter() - t0
    print(f"  {f.stem:30s} -> {pl_jsons[f.stem].shape}  ({elapsed:.3f}s)")

Loading all JSON files with Polars… dim_country → (212, 2) (0.000s) dim_index → (4, 5) (0.000s) eurostoxx50_ohlcv → (66355, 12) (0.107s) index_dim → (169, 26) (0.001s) index_performance → (5281, 15) (0.009s) oil20_ohlcv → (24738, 12) (0.032s) pulse → (40, 20) (0.000s) scores_daily → (466, 36) (0.002s) scores_quarterly → (170, 29) (0.001s) signals_daily → (466, 19) (0.001s) signals_quarterly → (177, 22) (0.001s) stoxxusa50_ohlcv → (65100, 12) (0.088s) trading_calendar → (29335, 11) (0.035s)

Reading Parquet Files

Pandas | read_parquet

Reads dim_country.parquet (212 rows × 2 columns) with default settings, then reads eurostoxx50_ohlcv.parquet with columns=["date", "symbol", "close"] — demonstrating column projection that returns 3 of 12 columns across 66 355 rows without loading the full schema.

df_pd_pq = pd.read_parquet(DATA_DIR / "dim_country.parquet")
print(f"Shape: {df_pd_pq.shape}")
print(f"Dtypes:\n{df_pd_pq.dtypes}")
display(df_pd_pq.head())

Shape: (212, 2) Dtypes: country_name object iso_alpha2 object dtype: object

country_nameiso_alpha2
0AfghanistanAF
1AlbaniaAL
2AlgeriaDZ
3American SamoaAS
4AndorraAD
# Read specific columns only (Parquet supports column projection)
df_pd_pq_cols = pd.read_parquet(
    DATA_DIR / "eurostoxx50_ohlcv.parquet",
    columns=["date", "symbol", "close"],
)
print(f"Shape (projected): {df_pd_pq_cols.shape}")
display(df_pd_pq_cols.head(3))

Shape (projected): (66355, 3)

datesymbolclose
02021-01-04ABI.BR57.21
12021-01-05ABI.BR57.18
22021-01-06ABI.BR58.77

Polars | read_parquet (Eager)

Reads dim_country.parquet with a native String schema (no coercion), then reads eurostoxx50_ohlcv.parquet selecting only date, symbol, and close — confirming Polars preserves native Parquet types (Date, String, Float64) without any post-load casting.

df_pl_pq = pl.read_parquet(DATA_DIR / "dim_country.parquet")
print(f"Shape: {df_pl_pq.shape}")
print(f"Schema: {df_pl_pq.schema}")
display(df_pl_pq.head())

Shape: (212, 2) Schema: Schema({‘country_name’: String, ‘iso_alpha2’: String})

country_nameiso_alpha2
strstr
AfghanistanAF
AlbaniaAL
AlgeriaDZ
American SamoaAS
AndorraAD
# Polars read_parquet with column selection
df_pl_pq_cols = pl.read_parquet(
    DATA_DIR / "eurostoxx50_ohlcv.parquet",
    columns=["date", "symbol", "close"],
)
print(f"Shape (projected): {df_pl_pq_cols.shape}")
display(df_pl_pq_cols.head(3))

Shape (projected): (66355, 3)

datesymbolclose
datestrf64
2021-01-04ABI.BR57.21
2021-01-05ABI.BR57.18
2021-01-06ABI.BR58.77

Polars | scan_parquet (Lazy)

scan_parquet() reads only Parquet metadata — no row data is loaded until .collect(). Combined with .filter() and .select(), the optimizer pushes both predicates and projections down to the Parquet reader.

Scans eurostoxx50_ohlcv.parquet lazily, then filters to ADYEN.AS, selects two columns, sorts descending, and collects only 10 rows — confirming the optimizer reads far less than the full 66 355-row file.

lf_pq = pl.scan_parquet(DATA_DIR / "eurostoxx50_ohlcv.parquet")
print(f"Type: {type(lf_pq)}")
print(f"Schema: {lf_pq.collect_schema()}")

Type: <class ‘polars.lazyframe.frame.LazyFrame’> Schema: Schema({‘id’: Int64, ‘symbol’: String, ‘date’: Date, ‘open’: Float64, ‘high’: Float64, ‘low’: Float64, ‘close’: Float64, ‘adj_close’: Float64, ‘volume’: Int64, ‘dividends’: Float64, ‘stock_splits’: Float64, ‘is_filled’: Boolean})

# Lazy scan with predicate pushdown and projection pushdown
result_pq = (
    lf_pq
    .filter(pl.col("symbol") == "ADYEN.AS")
    .select("date", "close")
    .sort("date", descending=True)
    .head(10)
    .collect()
)
display(Markdown("### Lazy parquet scan -> filtered, sorted, collected"))
display(result_pq)

Lazy parquet scan → filtered, sorted, collected

dateclose
datef64
2026-03-12925.7
2026-03-11926.5
2026-03-10935.0
2026-03-09942.7
2026-03-06930.4
2026-03-05931.5
2026-03-04957.6
2026-03-03949.1
2026-03-02965.7
2026-02-27994.8

Pandas | Bulk-load all Parquet files with timing

Loops over 13 Parquet files and loads each with pd.read_parquet(), printing per-file shape and elapsed time — confirming Parquet reads are sub-10 ms even for the largest files, with eurostoxx50_ohlcv (66 355 rows) loading in 0.006 s.

# Load ALL parquet files with Pandas
print("Loading all Parquet files with Pandas...")
pd_pqs = {}
for f in pq_files:
    t0 = time.perf_counter()
    pd_pqs[f.stem] = pd.read_parquet(f)
    elapsed = time.perf_counter() - t0
    print(f"  {f.stem:30s} -> {pd_pqs[f.stem].shape}  ({elapsed:.3f}s)")

Loading all Parquet files with Pandas… dim_country → (212, 2) (0.002s) dim_index → (4, 5) (0.001s) eurostoxx50_ohlcv → (66355, 12) (0.006s) index_dim → (169, 26) (0.005s) index_performance → (5281, 15) (0.002s) oil20_ohlcv → (24738, 12) (0.003s) pulse → (40, 20) (0.002s) scores_daily → (466, 36) (0.002s) scores_quarterly → (170, 29) (0.002s) signals_daily → (466, 19) (0.001s) signals_quarterly → (177, 22) (0.001s) stoxxusa50_ohlcv → (65100, 12) (0.005s) trading_calendar → (29335, 11) (0.003s)

Polars | Bulk-load all Parquet files with timing

Loops over the same 13 Parquet files with pl.read_parquet(), printing per-file shape and elapsed time — showing Polars at or below Pandas speeds, with eurostoxx50_ohlcv (66 355 rows) loading in 0.004 s vs Pandas’ 0.006 s.

# Load ALL parquet files with Polars
print("Loading all Parquet files with Polars...")
pl_pqs = {}
for f in pq_files:
    t0 = time.perf_counter()
    pl_pqs[f.stem] = pl.read_parquet(f)
    elapsed = time.perf_counter() - t0
    print(f"  {f.stem:30s} -> {pl_pqs[f.stem].shape}  ({elapsed:.3f}s)")

Loading all Parquet files with Polars… dim_country → (212, 2) (0.001s) dim_index → (4, 5) (0.001s) eurostoxx50_ohlcv → (66355, 12) (0.004s) index_dim → (169, 26) (0.001s) index_performance → (5281, 15) (0.001s) oil20_ohlcv → (24738, 12) (0.002s) pulse → (40, 20) (0.001s) scores_daily → (466, 36) (0.001s) scores_quarterly → (170, 29) (0.001s) signals_daily → (466, 19) (0.001s) signals_quarterly → (177, 22) (0.001s) stoxxusa50_ohlcv → (65100, 12) (0.004s) trading_calendar → (29335, 11) (0.001s)

Parameter Deep-Dives

Pandas | dtype override for column types at read time

Pandas uses dtype= to override column types at read time. Polars uses schema_overrides= for the same purpose. Both accept a dict mapping column names to types.

Reads pulse.csv with dtype={"ticker": "category"} — demonstrating that Pandas accepts a single-column override dict, while the remaining columns are inferred automatically.

df_dtype_pd = pd.read_csv(
    DATA_DIR / "pulse.csv",
    dtype={
        "ticker": "category",
    },
)
print("Pandas dtypes with category override:")
print(df_dtype_pd.dtypes)
display(df_dtype_pd.head(3))

Pandas dtypes with category override: id int64 _index object _ingested_at object symbol object timestamp object current_price float64 open_price float64 day_high float64 day_low float64 previous_close float64 price_change float64 price_change_pct float64 bid float64 ask float64 bid_size float64 ask_size float64 spread float64 current_volume int64 average_volume_10day int64 volume_ratio float64 dtype: object

id_index_ingested_atsymboltimestampcurrent_priceopen_priceday_highday_lowprevious_closeprice_changeprice_change_pctbidaskbid_sizeask_sizespreadcurrent_volumeaverage_volume_10dayvolume_ratio
020192euro_stoxx_502026-03-12 12:50:13.639560BMW.DE2026-03-12 13:49:5480.4079.081.1677.9080.82-0.42-0.519780.3880.520.00.00.1477068112098190.6370
120193euro_stoxx_502026-03-12 12:50:13.639560RHM.DE2026-03-12 13:49:551551.001536.01588.001535.001520.5030.502.00591551.501552.00267.045.00.501596332949730.5412
220194euro_stoxx_502026-03-12 12:50:13.639560BAS.DE2026-03-12 13:49:5547.6746.348.1045.9646.311.362.936747.6847.711393.0165.00.03151280040891340.3700

Polars | schema_overrides for column types at read time

Reads pulse.csv with schema_overrides={"ticker": pl.Categorical} — showing Polars’ dedicated parameter name for the same column-type-override concept, with the rest of the 20-column schema inferred automatically.

df_dtype_pl = pl.read_csv(
    DATA_DIR / "pulse.csv",
    schema_overrides={
        "ticker": pl.Categorical,
    },
)
print("Polars schema with overrides:")
print(df_dtype_pl.schema)
display(df_dtype_pl.head(3))

Polars schema with overrides: Schema({‘id’: Int64, ‘_index’: String, ‘_ingested_at’: String, ‘symbol’: String, ‘timestamp’: String, ‘current_price’: Float64, ‘open_price’: Float64, ‘day_high’: Float64, ‘day_low’: Float64, ‘previous_close’: Float64, ‘price_change’: Float64, ‘price_change_pct’: Float64, ‘bid’: Float64, ‘ask’: Float64, ‘bid_size’: Float64, ‘ask_size’: Float64, ‘spread’: Float64, ‘current_volume’: Int64, ‘average_volume_10day’: Int64, ‘volume_ratio’: Float64})

id_index_ingested_atsymboltimestampcurrent_priceopen_priceday_highday_lowprevious_closeprice_changeprice_change_pctbidaskbid_sizeask_sizespreadcurrent_volumeaverage_volume_10dayvolume_ratio
i64strstrstrstrf64f64f64f64f64f64f64f64f64f64f64f64i64i64f64
20192euro_stoxx_502026-03-12 12:50:13.639560BMW.DE2026-03-12 13:49:5480.479.081.1677.980.82-0.42-0.519780.3880.520.00.00.1477068112098190.637
20193euro_stoxx_502026-03-12 12:50:13.639560RHM.DE2026-03-12 13:49:551551.01536.01588.01535.01520.530.52.00591551.51552.0267.045.00.51596332949730.5412
20194euro_stoxx_502026-03-12 12:50:13.639560BAS.DE2026-03-12 13:49:5547.6746.348.145.9646.311.362.936747.6847.711393.0165.00.03151280040891340.37

Pandas | na_values parameter for null sentinel recognition

Pandas recognises many null sentinels by default (NA, N/A, null, empty string). You can extend with na_values=. Polars uses null_values= for the same purpose.

Reads scores_daily.csv with na_values=["", "NA", "N/A", "null", "-"] and counts nulls per column — revealing that 5 financial metric columns (pe_zscore, pb_zscore, ev_ebitda_zscore, yield_zscore, recommendation_mean) contain missing values.

df_null_pd = pd.read_csv(
    DATA_DIR / "scores_daily.csv",
    na_values=["", "NA", "N/A", "null", "-"],
)
null_counts_pd = df_null_pd.isnull().sum()
print("Pandas null counts per column:")
display(null_counts_pd[null_counts_pd > 0])

Pandas null counts per column:

pe_zscore 3 pb_zscore 6 ev_ebitda_zscore 71 yield_zscore 35 recommendation_mean 14 dtype: int64

Polars | null_values parameter for null sentinel recognition

Reads the same scores_daily.csv with Polars null_values= and calls .null_count() — displaying a wide 1-row × 36-column DataFrame of per-column null counts, confirming the same 5 columns with identical counts as Pandas.

df_null_pl = pl.read_csv(
    DATA_DIR / "scores_daily.csv",
    null_values=["", "NA", "N/A", "null", "-"],
)
null_counts_pl = df_null_pl.null_count()
print("Polars null counts per column:")
display(null_counts_pl)

Polars null counts per column:

id_indexsymbolscore_datesectorpe_zscorepb_zscoreev_ebitda_zscoreyield_zscorerelative_value_scorerelative_value_rankrelative_strengthsma_50_ratiosma_200_ratiodist_from_52w_highmomentum_scoremomentum_rankimplied_upsiderecommendation_meanprice_falling_analysts_bullishsentiment_scoresentiment_rankcomposite_scorecomposite_rank_scored_atsma_30_closesma_90_closemarket_capindex_weightshort_namecountrycurrent_priceday_change_pctfive_day_change_pctytd_change_pctcurrency
u32u32u32u32u32u32u32u32u32u32u32u32u32u32u32u32u32u32u32u32u32u32u32u32u32u32u32u32u32u32u32u32u32u32u32u32
000003671350000000001400000000000000000

Pandas / Polars | separator parameter: sep vs separator

Pandas uses sep= while Polars uses separator=. Both default to comma.

Reads dim_index.csv with an explicit sep="," in Pandas and separator="," in Polars, confirming both produce a (4, 5) result — demonstrating the parameter naming difference between the two libraries.

df_sep_pd = pd.read_csv(DATA_DIR / "dim_index.csv", sep=",")
print(f"Pandas with explicit sep=',' -> shape {df_sep_pd.shape}")
 
df_sep_pl = pl.read_csv(DATA_DIR / "dim_index.csv", separator=",")
print(f"Polars with explicit separator=',' -> shape {df_sep_pl.shape}")
 
# Note: Pandas uses 'sep', Polars uses 'separator'
display(df_sep_pd.head(3))

Pandas with explicit sep=’,’ → shape (4, 5) Polars with explicit separator=’,’ → shape (4, 5)

index_keydisplay_namefile_prefixcolorcurrency
0euro_stoxx_50Euro Stoxx 50eurostoxx50#4285F4€
1oil_20Oil & Gas 20oil20#D4A017$
2stoxx_asia_50STOXX Asia/Pacific 50stoxxasia50#EF5350NaN

Writing Data

Both Pandas and Polars write to CSV, JSON, and Parquet. Note the API naming difference: Pandas uses .to_csv() / .to_json() / .to_parquet(), while Polars uses .write_csv() / .write_json() / .write_parquet().

Pandas / Polars | Writing CSV: to_csv vs write_csv

Reads scores_quarterly.parquet and writes it as CSV via Pandas .to_csv(index=False) and Polars .write_csv() — comparing output sizes (49.0 KB vs 49.3 KB) and confirming both produce valid CSV files.

OUT_DIR = Path("../data/_output")
OUT_DIR.mkdir(exist_ok=True)
 
# Pandas to_csv
df_pd_pq = pd.read_parquet(DATA_DIR / "scores_quarterly.parquet")
csv_path_pd = OUT_DIR / "scores_quarterly_pandas.csv"
df_pd_pq.to_csv(csv_path_pd, index=False)
print(f"Pandas CSV written: {csv_path_pd.stat().st_size / 1024:.1f} KB")
 
# Polars to_csv (called write_csv)
df_pl_pq2 = pl.read_parquet(DATA_DIR / "scores_quarterly.parquet")
csv_path_pl = OUT_DIR / "scores_quarterly_polars.csv"
df_pl_pq2.write_csv(csv_path_pl)
print(f"Polars CSV written: {csv_path_pl.stat().st_size / 1024:.1f} KB")

Pandas CSV written: 49.0 KB Polars CSV written: 49.3 KB

Pandas / Polars | Writing JSON: to_json vs write_json

Writes the same scores_quarterly DataFrame as JSON via Pandas .to_json(orient="records", indent=2) and Polars .write_json() — showing Polars produces a more compact file (128.7 KB vs 143.6 KB) due to different default formatting.

# Pandas to_json
json_path_pd = OUT_DIR / "scores_quarterly_pandas.json"
df_pd_pq.to_json(json_path_pd, orient="records", indent=2)
print(f"Pandas JSON written: {json_path_pd.stat().st_size / 1024:.1f} KB")
 
# Polars write_json
json_path_pl = OUT_DIR / "scores_quarterly_polars.json"
df_pl_pq2.write_json(json_path_pl)
print(f"Polars JSON written: {json_path_pl.stat().st_size / 1024:.1f} KB")

Pandas JSON written: 143.6 KB Polars JSON written: 128.7 KB

Pandas / Polars | Writing Parquet: to_parquet vs write_parquet

Writes scores_quarterly as Parquet via both libraries — Polars produces a smaller file (23.9 KB vs 33.6 KB) due to more aggressive compression defaults, while both produce Arrow-compatible Parquet files.

# Pandas to_parquet
pq_path_pd = OUT_DIR / "scores_quarterly_pandas.parquet"
df_pd_pq.to_parquet(pq_path_pd, index=False)
print(f"Pandas Parquet written: {pq_path_pd.stat().st_size / 1024:.1f} KB")
 
# Polars write_parquet
pq_path_pl = OUT_DIR / "scores_quarterly_polars.parquet"
df_pl_pq2.write_parquet(pq_path_pl)
print(f"Polars Parquet written: {pq_path_pl.stat().st_size / 1024:.1f} KB")

Pandas Parquet written: 33.6 KB Polars Parquet written: 23.9 KB

Pandas | Compare output file sizes across formats and libraries

Globs all six output files from the _output directory and builds a name-to-size DataFrame — summarising CSV, JSON, and Parquet sizes for both Pandas and Polars side by side.

# Compare output file sizes
display(Markdown("### Output file size comparison"))
out_files = sorted(OUT_DIR.glob("scores_quarterly_*"))
rows = []
for f in out_files:
    rows.append({"file": f.name, "size_KB": round(f.stat().st_size / 1024, 1)})
display(pd.DataFrame(rows))

Output file size comparison

filesize_KB
0scores_quarterly_pandas.csv49.0
1scores_quarterly_pandas.json143.6
2scores_quarterly_pandas.parquet33.6
3scores_quarterly_polars.csv49.3
4scores_quarterly_polars.json128.7
5scores_quarterly_polars.parquet23.9

Python | Clean up output directory

Deletes the _output directory and all six generated files (CSV, JSON, Parquet × 2 libraries) created in the Writing Data section — keeping the working directory clean after the write demonstration.

# Cleanup output directory
shutil.rmtree(OUT_DIR)
print(f"Cleaned up {OUT_DIR}")

Cleaned up ..\data_output

Lazy Scanning vs Eager Reading

Polars’ scan_* functions return a LazyFrame that does not read data until .collect() is called. The query optimizer rewrites the plan for efficiency through three key mechanisms: predicate pushdown (filters applied at the file level), projection pushdown (only needed columns are read), and common subexpression elimination (avoid redundant work). Eager read_* loads everything into memory immediately.

When to use lazy vs eager

Use lazy (scan_csv, scan_parquet) when you only need a subset of rows or columns — the optimizer avoids reading unnecessary data. Use eager (read_csv, read_parquet) when you need the full dataset or when the file is small enough that optimization overhead outweighs savings.


flowchart LR
    subgraph Eager["Eager: read_parquet"]
        E1["Read ALL rows<br/>and columns"] --> E2["Filter in<br/>memory"] --> E3["Select<br/>columns"]
    end
    subgraph Lazy["Lazy: scan_parquet"]
        L1["Build<br/>query plan"] --> L2["Optimizer:<br/>pushdown"] --> L3["Read ONLY<br/>needed data"]
    end
    style Eager fill:#292e42,stroke:#565f89
    style Lazy fill:#1a1b26,stroke:#565f89

Polars | Eager Parquet read — full file load into memory

Reads all 66 355 rows × 12 columns of eurostoxx50_ohlcv.parquet into memory eagerly and records the elapsed time — establishing the baseline for comparison with the lazy filtered read below.

# Eager: reads entire file into memory
t0 = time.perf_counter()
df_eager = pl.read_parquet(DATA_DIR / "eurostoxx50_ohlcv.parquet")
eager_time = time.perf_counter() - t0
print(f"Eager read: {df_eager.shape}, {eager_time:.4f}s")

Eager read: (66355, 12), 0.0044s

Polars | Lazy Parquet scan with filter and collect

Scans the same file lazily, filters to ADYEN.AS, selects two columns, and collects — resulting in a 1 331-row result in 0.0018 s vs 0.0044 s for the eager full read, showing ~2.5× speedup from predicate + projection pushdown.

# Lazy: scan + filter + collect (only reads what's needed)
t0 = time.perf_counter()
df_lazy = (
    pl.scan_parquet(DATA_DIR / "eurostoxx50_ohlcv.parquet")
    .filter(pl.col("symbol") == "ADYEN.AS")
    .select("date", "close")
    .collect()
)
lazy_time = time.perf_counter() - t0
print(f"Lazy scan+filter+collect: {df_lazy.shape}, {lazy_time:.4f}s")
print(f"\nLazy was ~{eager_time / max(lazy_time, 0.0001):.1f}x vs eager for this filtered query")

Lazy scan+filter+collect: (1331, 2), 0.0018s

Lazy was ~2.5x vs eager for this filtered query

Polars | Explain optimized query plan

Builds a lazy plan with filter, column selection, and sort, then calls .explain() — printing the optimized execution plan bottom-up, confirming PROJECT 4/12 COLUMNS and predicate pushdown into the Parquet scan.

# Explain the query plan
plan = (
    pl.scan_parquet(DATA_DIR / "eurostoxx50_ohlcv.parquet")
    .filter(pl.col("symbol") == "ADYEN.AS")
    .select("date", "close", "volume")
    .sort("date")
)
print("=== Optimized Query Plan ===")
print(plan.explain())
=== Optimized Query Plan ===
SORT BY [col("date")]
  simple π 3/3 ["date", "close", "volume"]
    Parquet SCAN [../data/eurostoxx50_ohlcv.parquet]
    PROJECT 4/12 COLUMNS
    SELECTION: [(col("symbol")) == ("ADYEN.AS")]
    ESTIMATED ROWS: 66355

Reading the plan bottom-up: Parquet SCAN reads the file. PROJECT 4/12 COLUMNS means only 4 of 12 columns are loaded (projection pushdown — date, symbol, close, volume; symbol is needed for the filter). SELECTION shows the predicate pushed down to the scan. simple π 3/3 is the final projection that drops symbol after filtering. SORT BY sorts the result.

Polars | Lazy vs eager CSV scan comparison

Compares read_csv (eager, 24 738 rows) against scan_csv + filter + collect on oil20_ohlcv.csv — demonstrating that lazy CSV scanning with a symbol filter can be faster than loading the full file when only a subset of rows is needed.

# Lazy scan_csv comparison
t0 = time.perf_counter()
df_csv_eager = pl.read_csv(DATA_DIR / "oil20_ohlcv.csv", try_parse_dates=True)
csv_eager_time = time.perf_counter() - t0
 
t0 = time.perf_counter()
df_csv_lazy = (
    pl.scan_csv(DATA_DIR / "oil20_ohlcv.csv", try_parse_dates=True)
    .filter(pl.col("symbol") == "CL=F")
    .select("date", "close")
    .collect()
)
csv_lazy_time = time.perf_counter() - t0
 
print(f"CSV eager: {df_csv_eager.shape} in {csv_eager_time:.4f}s")
print(f"CSV lazy+filter: {df_csv_lazy.shape} in {csv_lazy_time:.4f}s")

CSV eager: (24738, 12) in 0.0023s CSV lazy+filter: (0, 2) in 0.0034s

Format Comparison | Size and Speed

For a deeper look at when to choose Parquet, CSV, or JSON across the full data pipeline, see serialization-formats. The same Parquet I/O patterns shown here apply when loading data into BigQuery via data-loading-and-export.

Pandas / Polars | Read speed benchmark across CSV, JSON, and Parquet

Runs timed reads for 5 datasets × 3 formats with both Pandas and Polars, accumulating results into a 15-row benchmark DataFrame — showing Polars consistently faster (often 3–10×) on CSV and JSON for medium-to-large files.

# Benchmark read speed: CSV vs JSON vs Parquet for Pandas and Polars
benchmark_datasets = ["dim_country", "pulse", "scores_daily",
                      "index_performance", "oil20_ohlcv"]
results = []
 
for name in benchmark_datasets:
    for fmt, reader_pd, reader_pl in [
        ("csv",     lambda p: pd.read_csv(p),     lambda p: pl.read_csv(p, try_parse_dates=True)),
        # Add infer_schema_length=None to the Polars JSON reader
        ("json",    lambda p: pd.read_json(p),    lambda p: pl.read_json(p, infer_schema_length=None)),
        ("parquet", lambda p: pd.read_parquet(p), lambda p: pl.read_parquet(p)),
    ]:
        fpath = DATA_DIR / f"{name}.{fmt}"
        if not fpath.exists():
            continue
        size_kb = fpath.stat().st_size / 1024
 
        t0 = time.perf_counter()
        _ = reader_pd(fpath)
        pd_time = time.perf_counter() - t0
 
        t0 = time.perf_counter()
        _ = reader_pl(fpath)
        pl_time = time.perf_counter() - t0
 
        results.append({
            "dataset": name,
            "format": fmt,
            "size_KB": round(size_kb, 1),
            "pandas_sec": round(pd_time, 4),
            "polars_sec": round(pl_time, 4),
        })
 
bench_df = pd.DataFrame(results)
display(Markdown("### Read Speed Benchmark"))
display(bench_df)

Read Speed Benchmark

datasetformatsize_KBpandas_secpolars_sec
0dim_countrycsv2.80.00130.0011
1dim_countryjson13.00.00140.0003
2dim_countryparquet5.00.00140.0011
3pulsecsv6.80.00090.0009
4pulsejson20.80.00330.0003
5pulseparquet16.40.00180.0007
6scores_dailycsv236.60.00310.0035
7scores_dailyjson541.00.00820.0025
8scores_dailyparquet117.40.00230.0009
9index_performancecsv940.10.00660.0025
10index_performancejson2432.60.01740.0088
11index_performanceparquet344.90.00230.0010
12oil20_ohlcvcsv1849.10.01420.0022
13oil20_ohlcvjson6511.60.06200.0326
14oil20_ohlcvparquet882.40.00320.0023

Pandas | Pivot benchmark results to compare file sizes across formats

Pivots bench_df so rows are datasets and columns are formats (CSV, JSON, Parquet), then adds a parquet_vs_csv_% column — making it easy to see that Parquet achieves 37–50% of CSV size for larger datasets.

# Pivot to compare formats side-by-side for file size
size_pivot = bench_df.pivot_table(
    index="dataset", columns="format", values="size_KB", aggfunc="first"
)[["csv", "json", "parquet"]]
size_pivot["parquet_vs_csv_%"] = (
    (size_pivot["parquet"] / size_pivot["csv"] * 100).round(1)
)
display(Markdown("### File Size Comparison (KB)"))
display(size_pivot)

File Size Comparison (KB)

formatcsvjsonparquetparquet_vs_csv_%
dataset
dim_country2.813.05.0178.6
index_performance940.12432.6344.936.7
oil20_ohlcv1849.16511.6882.447.7
pulse6.820.816.4241.2
scores_daily236.6541.0117.449.6

Gotchas and Tips

Pandas | Date parsing: parse_dates parameter in read_csv

  • Pandas: use parse_dates=["col"] in read_csv; JSON dates often

need pd.to_datetime() after loading.

  • Polars: use try_parse_dates=True in read_csv; Parquet stores

date types natively.

  • Gotcha: Pandas may silently parse dates as strings if the format is

ambiguous. Always verify dtypes after loading.

Reads trading_calendar.csv twice — once without and once with parse_dates=["date"] — showing the date column resolves as object by default and as datetime64[ns] when parsing is enabled.

# Pandas: dates in CSV may need explicit parsing
df_dates = pd.read_csv(DATA_DIR / "trading_calendar.csv")
print(f"date column dtype WITHOUT parse_dates: {df_dates['date'].dtype}")
 
df_dates2 = pd.read_csv(DATA_DIR / "trading_calendar.csv", parse_dates=["date"])
print(f"date column dtype WITH parse_dates:    {df_dates2['date'].dtype}")

date column dtype WITHOUT parse_dates: object date column dtype WITH parse_dates: datetime64[ns]

Pandas | Chunked reading for large CSV files with chunksize

  • Parquet supports column projection - read only the columns you need.
  • Polars lazy scans avoid loading entire files.
  • Pandas read_csv with chunksize returns an iterator for large files.

Reads eurostoxx50_ohlcv.csv in 10 000-row chunks via the chunksize iterator, accumulating total_rows — demonstrating that Pandas can process files larger than memory without loading everything at once.

# Pandas chunked reading
chunk_iter = pd.read_csv(DATA_DIR / "eurostoxx50_ohlcv.csv", chunksize=10_000)
total_rows = 0
for chunk in chunk_iter:
    total_rows += len(chunk)
print(f"Total rows via chunked reading: {total_rows:,}")

Total rows via chunked reading: 66,355

Pandas | Index Handling in CSV output

Pandas’ .to_csv() includes the index by default — always pass index=False when writing data intended for other systems.

Writes a 2-row DataFrame to a StringIO buffer twice — once with the default index and once with index=False — printing both outputs to show the unwanted leading row-number column that appears in default CSV output.

# Pandas default to_csv includes the index
buf = io.StringIO()
pd.DataFrame({"a": [1, 2]}).to_csv(buf)
print("With index (default):")
print(buf.getvalue())
 
buf2 = io.StringIO()
pd.DataFrame({"a": [1, 2]}).to_csv(buf2, index=False)
print("Without index:")
print(buf2.getvalue())

With index (default): ,a 0,1 1,2

Without index: a 1 2

Pandas | Memory savings: string vs category dtype for low-cardinality columns

  • For columns with low cardinality (e.g. tickers, country codes),

use category (Pandas) or Categorical (Polars) to save memory.

  • Set dtypes at read time for best performance.

Reads only the symbol column from eurostoxx50_ohlcv.csv twice — once as object and once as category — and compares memory usage, demonstrating a 98% reduction (3 569 KB → 70 KB) from category encoding.

# Memory comparison: string vs category in Pandas
df_str = pd.read_csv(DATA_DIR / "eurostoxx50_ohlcv.csv", usecols=["symbol"])
df_cat = pd.read_csv(DATA_DIR / "eurostoxx50_ohlcv.csv", usecols=["symbol"],
                      dtype={"symbol": "category"})
 
mem_str = df_str.memory_usage(deep=True).sum() / 1024
mem_cat = df_cat.memory_usage(deep=True).sum() / 1024
print(f"String dtype memory:   {mem_str:,.1f} KB")
print(f"Category dtype memory: {mem_cat:,.1f} KB")
print(f"Savings: {(1 - mem_cat/mem_str)*100:.1f}%")
String dtype memory:   3,569.2 KB
Category dtype memory: 69.7 KB
Savings: 98.0%

Summary Comparison | Reading & Writing

Pandas / Polars | Reading and writing API comparison table

Builds a 15-row comparison table mapping each I/O operation to its Pandas and Polars equivalents, highlighting key differences such as parameter naming (dtype= vs schema_overrides=), missing lazy-read support in Pandas, and index handling in CSV output.

comparison = [
    ["Read CSV",           "pd.read_csv()",        "pl.read_csv()",          "Both excellent"],
    ["Read JSON",          "pd.read_json()",       "pl.read_json()",         "Polars stricter on schema"],
    ["Read Parquet",       "pd.read_parquet()",    "pl.read_parquet()",      "Both use Arrow under the hood"],
    ["Lazy CSV",           "N/A (use chunksize)",  "pl.scan_csv()",          "Polars only"],
    ["Lazy NDJSON",        "N/A",                  "pl.scan_ndjson()",       "Polars only; needs NDJSON format"],
    ["Lazy Parquet",       "N/A",                  "pl.scan_parquet()",      "Polars only; best lazy format"],
    ["Write CSV",          ".to_csv()",            ".write_csv()",           "Pandas writes index by default"],
    ["Write JSON",         ".to_json()",           ".write_json()",          "Different default orientations"],
    ["Write Parquet",      ".to_parquet()",        ".write_parquet()",       "Both produce valid Parquet"],
    ["dtype override",     "dtype={...}",          "dtypes={...}",          "Param name differs"],
    ["Schema override",    "dtype={...}",          "schema_overrides={...}", "Polars has dedicated param"],
    ["Null values",        "na_values=[...]",      "null_values=[...]",      "Param name differs"],
    ["Separator",          "sep=','",              "separator=','",          "Param name differs"],
    ["Column projection",  "usecols=[...]",        "columns=[...]",          "Parquet: both support this"],
    ["Predicate pushdown", "N/A",                  "LazyFrame.filter()",     "Polars only; major advantage"],
]
 
comp_df = pd.DataFrame(
    comparison, columns=["Operation", "Pandas", "Polars", "Notes"]
)
display(Markdown("### Pandas vs Polars - Reading & Writing Comparison"))
display(comp_df.style.set_properties(**{"text-align": "left"}).hide(axis="index")) # type: ignore

Pandas vs Polars - Reading & Writing Comparison

OperationPandasPolarsNotes
Read CSVpd.read_csv()pl.read_csv()Both excellent
Read JSONpd.read_json()pl.read_json()Polars stricter on schema
Read Parquetpd.read_parquet()pl.read_parquet()Both use Arrow under the hood
Lazy CSVN/A (use chunksize)pl.scan_csv()Polars only
Lazy NDJSONN/Apl.scan_ndjson()Polars only; needs NDJSON format
Lazy ParquetN/Apl.scan_parquet()Polars only; best lazy format
Write CSV.to_csv().write_csv()Pandas writes index by default
Write JSON.to_json().write_json()Different default orientations
Write Parquet.to_parquet().write_parquet()Both produce valid Parquet
dtype overridedtype={...}dtypes={...}Param name differs
Schema overridedtype={...}schema_overrides={...}Polars has dedicated param
Null valuesna_values=[...]null_values=[...]Param name differs
Separatorsep=','separator=','Param name differs
Column projectionusecols=[...]columns=[...]Parquet: both support this
Predicate pushdownN/ALazyFrame.filter()Polars only; major advantage

Key Takeaways | Reading & Writing

  1. Parquet is the best format for analytical workloads: smallest files,

fastest reads, native schema preservation.

  1. Polars lazy scanning (scan_csv, scan_parquet, scan_ndjson)

enables predicate and projection pushdown - only reads what you need.

  1. Parameter names differ between Pandas and Polars (sep vs

separator, dtype vs dtypes / schema_overrides, etc.).

  1. Always verify dtypes after loading CSV/JSON - both libraries may

guess wrong on dates, nulls, or mixed-type columns.

  1. For large files, prefer Parquet + Polars lazy for best performance.

Common Traps and Safe Patterns

Nulls upcast integer columns

Inserting a single NaN into a Pandas int64 column silently converts the entire column to float64. This changes the semantics of the data: identifiers become floats, equality checks become fragile, and downstream code can start comparing 1.0 to 1.

[!success] Use nullable integer dtypes

Create nullable integer columns with pd.Int64Dtype() and pd.NA, or load them that way at read time. In Polars, Arrow nulls preserve the integer dtype automatically, so the safe pattern is simply to keep the column typed as integer.

CSV inference is unreliable

Both pd.read_csv() and pl.read_csv() infer column types from the observed data. Dates can stay as strings, nullable integers can become floats, and mixed columns can degrade to object or a wider inferred type that was never part of the intended schema.

[!success] Declare read schemas explicitly

Pass dtype= or dtype_backend= plus explicit date parsing in Pandas, and schema_overrides= in Polars. Production reads should enter the pipeline with a declared schema, not a guessed one.

Object dtype kills vectorization

A Pandas object column falls back to Python-object semantics instead of native vectorized kernels. That makes operations slower, weakens type guarantees, and often hides mixed values until a later transformation fails.

[!success] Normalize dtypes immediately

Convert text columns to string or category, numeric columns to concrete numeric dtypes, and dates to datetime types as soon as the data is loaded. Early normalization keeps the rest of the pipeline on predictable, vectorized paths.

.values may share memory

Pandas .values can expose the underlying array storage instead of an isolated copy. Mutating that array can therefore mutate the originating Series or DataFrame unexpectedly, which is exactly the kind of side effect that spreads silently through notebook workflows.

[!success] Copy arrays before mutation

If you need an independent NumPy buffer, use .to_numpy(copy=True) or call .copy() before mutating. Treat .values as a low-level escape hatch for inspection, not as the default handoff for mutable downstream work.

Duplicate column names are legal

Pandas accepts duplicate column names without complaint. Selecting by name can then return multiple columns instead of one, and later merge, assign, or rename steps become ambiguous in ways that are difficult to debug.

[!success] Fail fast on duplicate columns

Check df.columns.is_unique after every load and rename boundary, and either deduplicate immediately or raise an error. Polars already enforces this at creation time; Pandas code should add the same discipline explicitly.

int64 overflow wraps silently

NumPy-backed integer arithmetic in Pandas wraps on overflow instead of raising. Once a value crosses np.iinfo(np.int64).max, the result can jump to the most negative int64 value with no warning at all.

[!success] Guard arithmetic near type limits

Validate numeric ranges before large accumulations, and promote overflow-prone counters to a wider or non-wrapping representation before the calculation. Do not rely on post hoc inspection to catch wrapped values after the fact.

Schema drift breaks assumptions

Upstream sources can rename, reorder, add, or remove columns without notice. Downstream DataFrame code may then fail loudly, or worse, continue running against the wrong fields because a similarly named replacement happened to exist.

[!success] Assert schemas at every boundary

After every file read, API ingest, or database query, check both column names and dtypes against the expected contract before continuing. Schema validation belongs at the input boundary, not after a broken transform has already propagated bad state.

Python Foundations and I/O Recommendations

  1. Use Polars for new projects — stricter type system, no index-related bugs, native lazy evaluation, and Arrow-backed immutability eliminate entire categories of Pandas footguns.
  2. Always specify dtypes on CSV/JSON reads — never rely on inference for production data. Pass dtype={} (Pandas) or dtypes={} / schema_overrides={} (Polars) explicitly.
  3. Prefer Parquet over CSV — Parquet preserves schema, compresses 5–10x smaller, reads 5–50x faster, and supports column projection and predicate pushdown in Polars lazy mode.
  4. Validate schemas after every I/O boundary — after reading a file, API response, or database query, assert that column names and types match expectations before proceeding.
  5. Use nullable dtypes in Pandas — pd.Int64Dtype(), pd.StringDtype(), and pd.BooleanDtype() prevent silent type promotion from nulls.
  6. Avoid .apply() with Python lambdas — use vectorized expressions instead. In Polars, use .map_elements() only as a last resort (it drops to Python-level speed).
  7. Profile memory before scaling — use df.memory_usage(deep=True) (Pandas) or df.estimated_size() (Polars) to verify that your data actually fits in memory before running expensive operations.
  8. Keep index usage minimal in Pandas — if you must use Pandas, prefer .reset_index() early and use column-based operations. This reduces alignment bugs and makes migration to Polars easier.

Troubleshooting and failure modes

SymptomLikely causeFix
Column dtype is float64 but data should be integersNaN in the column forced promotionUse pd.Int64Dtype() or clean nulls before casting
KeyError on column selectionColumn name has leading/trailing whitespacedf.columns = df.columns.str.strip()
.merge() produces more rows than expectedDuplicate keys in one or both sides (many-to-many)Deduplicate keys or use validate="one_to_one" / validate="many_to_one"
Polars SchemaError on DataFrame creationDuplicate column names or type mismatchCheck for duplicate keys in the source dict; verify types match the declared schema
OutOfMemoryError on read_csv()File exceeds available RAMUse pl.scan_csv() with lazy evaluation, or read in chunks with pd.read_csv(chunksize=...)
Parquet read fails with codec errorParquet file uses a compression codec not installed (snappy, zstd, lz4)Install the missing codec: pip install pyarrow[snappy] or pip install python-snappy
.to_csv() writes an unexpected index columnPandas writes the index by defaultPass index=False to .to_csv()
Polars ComputeError: NaN is not comparableUsing NaN in a filter or sort on a float columnReplace NaN with null: df.with_columns(pl.col("x").fill_nan(None))
Arithmetic on two Pandas DataFrames produces all NaNIndex misalignment — the two DataFrames have different indexes.reset_index() both before operating, or use .values for positional arithmetic
String column shows as object with poor performancePandas defaulted to object dtype for stringsConvert to pd.StringDtype(): df["col"] = df["col"].astype("string")

Graph View

Backlinks