06 - Lazy API and Performance - C#

20 min read

Lazy API and Performance - C#

Quote

“Premature optimization is the root of all evil.”

— Donald Knuth, Structured Programming with go to Statements (1974)

“The First Rule of Program Optimization: Don’t do it. The Second Rule of Program Optimization (for experts only): Don’t do it yet.”

— Michael A. Jackson, Principles of Program Design (1975)

C# Lazy API and Performance Setup

Setup | Suppress compiler warnings

Silences Roslyn diagnostic warnings in the .NET Interactive kernel to keep notebook output clean. This is standard boilerplate for .NET notebooks — do not modify.

Kernel warning suppression.

using System.Reflection;
using Microsoft.DotNet.Interactive;
using Microsoft.DotNet.Interactive.CSharp;
 
var csharpKernel = (CSharpKernel)Kernel.Root.FindKernelByName("csharp");
var optionsField = typeof(CSharpKernel).GetField("_scriptOptions",
    BindingFlags.NonPublic | BindingFlags.Instance);
 
var scriptOptions = optionsField.GetValue(csharpKernel);
var withWarningLevel = scriptOptions.GetType().GetMethod("WithWarningLevel");
var newOptions = withWarningLevel.Invoke(scriptOptions, new object[] { 0 });
optionsField.SetValue(csharpKernel, newOptions);
No visible output.

Setup | Install NuGet packages and configure formatters

Loads Polars.NET 0.4.0 and its native Windows x64 runtime via NuGet. Registers custom HTML formatters so DataFrame and Series values render as tables in notebook output. Sets DATA to the shared dataset directory.

Package and formatter setup.

#r "nuget: Polars.NET, 0.4.0"
#r "nuget: Polars.NET.Native.win-x64, 0.4.0"
 
using System.IO;
using System.Linq;
using System.Diagnostics;
using Polars.CSharp;
using static Polars.CSharp.Polars;
using Microsoft.DotNet.Interactive.Formatting;
 
Formatter.Register<DataFrame>((df, writer) =>
{
    var html = df.ToHtml();
    html = System.Text.RegularExpressions.Regex.Replace(html, @"(>|>)(.+?)(<|<)", @"$1$2$3");
    html = System.Text.RegularExpressions.Regex.Replace(html, @">""(.+?)""<", @">$1<");
    var css = """
        """;
    writer.Write(css + html);
}, "text/html");
Formatter.Register<Polars.CSharp.Series>((s, writer) =>
    writer.Write($"<pre style='font-size:14px'>{s}</pre>"), "text/html");
 
var DATA = Path.Combine("..", "data");
Console.WriteLine($"Data directory: {Path.GetFullPath(DATA)}");
Data directory: c:\Users\aperi\DEV\LANG\data

Lazy Fundamentals

Polars has two execution modes:

  • Eager — operations execute immediately and return a DataFrame.
  • Lazy — operations build a query plan (a LazyFrame) that only executes when you call .Collect().

The lazy API lets Polars optimize the entire query plan before touching any data: reordering filters, eliminating unused columns, and pushing predicates down to the file scanner.

Eager load of large files exhausts memory

DataFrame.ReadParquet(path) loads the entire file into RAM immediately. For files larger than available memory, use LazyFrame.ScanParquet(path) and chain .Filter() / .Select() before .Collect() — Polars will only materialize the rows and columns you actually need.


flowchart LR
    A["ScanParquet(path)"] -->|"LazyFrame — no data read"| B["Build query plan"]
    B --> C[".Filter(Col(...))"]
    C --> D[".Select(Col(...))"]
    D --> E[".Sort / .GroupBy<br/>.WithColumns"]
    E -->|"Optimizer rewrites plan"| F[".Collect()"]
    F --> G["DataFrame<br/>(materialized)"]

    style A fill:#1f2335,stroke:#7aa2f7,color:#c0caf5
    style B fill:#1f2335,stroke:#7aa2f7,color:#c0caf5
    style C fill:#1f2335,stroke:#e0af68,color:#c0caf5
    style D fill:#1f2335,stroke:#e0af68,color:#c0caf5
    style E fill:#1f2335,stroke:#e0af68,color:#c0caf5
    style F fill:#1f2335,stroke:#f7768e,color:#f7768e
    style G fill:#1f2335,stroke:#9ece6a,color:#9ece6a

Scan, Convert, and Collect

Polars.NET | Scan Parquet (lazy)

LazyFrame.ScanParquet(path) registers the Parquet file in the query plan without reading any data. Schema is inferred from file metadata. Use this as the entry point for all lazy pipelines — it enables predicate and projection pushdown so only the rows and columns you actually need are ever read from disk.

Lazy Parquet scan.

var parquetPath = Path.Combine(DATA, "eurostoxx50_ohlcv.parquet");
var lf = LazyFrame.ScanParquet(parquetPath);
 
Console.WriteLine($"Type: {lf.GetType().Name}");
Console.WriteLine("No data has been loaded yet — just a query plan.");
Type: LazyFrame
No data has been loaded yet — just a query plan.

Polars.NET | Eager to Lazy conversion

Call .Lazy() on an existing DataFrame to convert it to a LazyFrame. The conversion is free — no data is copied. Use this when you have already read data eagerly (e.g., from a CSV) but want to chain further operations with lazy optimization before collecting.

Eager to lazy conversion.

var dfEager = DataFrame.ReadCsv(Path.Combine(DATA, "eurostoxx50_ohlcv.csv"));
Console.WriteLine($"Eager DataFrame shape: {dfEager.Shape}");
 
var lfFromEager = dfEager.Lazy();
Console.WriteLine($"LazyFrame type: {lfFromEager.GetType().Name}");
Console.WriteLine("Eager -> Lazy conversion is free (no copy).");
Eager DataFrame shape: (66355, 12)
LazyFrame type: LazyFrame
Eager -> Lazy conversion is free (no copy).

Polars.NET | Collect

.Collect() is the execution trigger

Nothing runs until you call .Collect(). A LazyFrame with filters, selects, and sorts is just a description of work — no computation happens. If you forget .Collect(), you will hold a LazyFrame with no results. For large datasets, call .Head(100).Collect() first to validate the plan before collecting the full result.

.Collect() executes the optimized query plan and materializes the result as a DataFrame. This is the only point where data moves — Polars reads the file, applies filters, and evaluates expressions.

Collect the lazy frame.

var result = lf.Collect();
Console.WriteLine($"Collected shape: {result.Shape}");
 
result.Head(5)
Collected shape: (66355, 12)
idsymboldateopenhighlowcloseadj_closevolumedividendsstock_splitsis_filled
21160ABI.BR2021-01-0458.1558.8556.7857.2153.5761151393700false
21161ABI.BR2021-01-0556.957.9856.7557.1853.548138272200false
21162ABI.BR2021-01-0657.9658.9457.3958.7755.037137020400false
21163ABI.BR2021-01-0758.6858.8657.8858.454.6905146991100false
21164ABI.BR2021-01-0858.1658.457.4357.8654.1848142868100false

Query Plan Inspection

Polars.NET | Explain: inspect the optimized query plan

lf.Explain() returns the optimized query plan as a string. Read it to verify that filters and column selections have been pushed into the scan. The plan shows PROJECT N/12 COLUMNS for projection pushdown and SELECTION: [...] for predicate pushdown.

Explain() availability in Polars.NET 0.4.0

Explain() may not be exposed in Polars.NET 0.4.0. The code below catches the exception gracefully. Regardless, Polars still applies all optimizations internally when you call .Collect() — you just cannot inspect the plan text in this version.

Query plan inspection.

try
{
    var lfExplain = LazyFrame.ScanParquet(Path.Combine(DATA, "eurostoxx50_ohlcv.parquet"));
    var plan = lfExplain.Explain();
    Console.WriteLine("Optimized query plan:");
    Console.WriteLine(plan);
}
catch (Exception ex)
{
    Console.WriteLine($"Explain() not available in Polars.NET 0.4.0: {ex.GetType().Name}");
    Console.WriteLine("Use .Collect() to execute — optimization still happens internally.");
}
Optimized query plan:
Parquet SCAN [../data/eurostoxx50_ohlcv.parquet]
PROJECT */12 COLUMNS
ESTIMATED ROWS: 66355

Query Optimization

When you build a lazy query, Polars applies automatic optimizations before execution:

  • Predicate pushdown — filters move as early as possible, even into the file scanner.
  • Projection pushdown — only columns that are actually used get read from disk.
  • Common subexpression elimination — duplicate computations are evaluated once.

These happen transparently. You write clear, readable code; Polars figures out the fastest plan.

Write readable code — let Polars optimize

Do not manually reorder filters to “help” Polars. Write the query in logical order (scan → filter → select → aggregate). Polars will rewrite the plan into the optimal execution order automatically, including pushing filters inside the file scanner.

Predicate and Projection Pushdown

Polars.NET | Predicate pushdown

A filter applied to a LazyFrame is pushed into the file scanner at optimization time. Polars passes the predicate to the Parquet reader, which skips row groups that cannot match — so only qualifying rows are ever deserialized into memory.

Predicate pushdown.

var lfFiltered = LazyFrame.ScanParquet(Path.Combine(DATA, "eurostoxx50_ohlcv.parquet"))
    .Filter(Col("symbol") == Lit("SAP.DE"));
 
var dfFiltered = lfFiltered.Collect();
Console.WriteLine($"Rows matching symbol='SAP.DE': {dfFiltered.Shape}");
 
dfFiltered.Head(5)
Rows matching symbol='SAP.DE': (1324, 12)
idsymboldateopenhighlowcloseadj_closevolumedividendsstock_splitsis_filled
5301SAP.DE2021-01-04108.1108.5104.78105.3297.0102292851500false
5302SAP.DE2021-01-05104.98106.2104.46105.0496.7523279888800false
5303SAP.DE2021-01-06105.14106.26103.6105.4897.1576301880200false
5304SAP.DE2021-01-07105.58105.7104.04104.5296.2734317614300false
5305SAP.DE2021-01-08105.14106.72105.04106.1897.8024306874400false

Polars.NET | Projection pushdown

A .Select() on a LazyFrame is pushed into the Parquet scanner — only the requested columns are read from disk. Parquet’s columnar format stores each column separately, so unselected columns are completely skipped. This is one of the largest memory and I/O savings in the Polars lazy API.

Projection pushdown.

var lfProjected = LazyFrame.ScanParquet(Path.Combine(DATA, "eurostoxx50_ohlcv.parquet"))
    .Select(Col("symbol"), Col("date"), Col("close"), Col("volume"));
 
var dfProjected = lfProjected.Collect();
Console.WriteLine($"Projected shape: {dfProjected.Shape} (only 4 of 12 columns read)");
 
dfProjected.Head(5)
Projected shape: (66355, 4) (only 4 of 12 columns read)
symboldateclosevolume
ABI.BR2021-01-0457.211513937
ABI.BR2021-01-0557.181382722
ABI.BR2021-01-0658.771370204
ABI.BR2021-01-0758.41469911
ABI.BR2021-01-0857.861428681

Lazy Pipeline Operations

Polars.NET | Combined filter, select, and sort

Chaining .Filter(), .Select(), and .Sort() on a LazyFrame builds a single query plan. Polars optimizes the full pipeline before any execution: the filter is pushed into the scan, only the four selected columns are read from disk, and the sort runs on the already-filtered result. Write the chain in the logical order that reads clearly — Polars handles the reordering.

Combined lazy pipeline.

var lfCombined = LazyFrame.ScanParquet(Path.Combine(DATA, "eurostoxx50_ohlcv.parquet"))
    .Filter(Col("volume") > Lit(5_000_000))
    .Select(Col("symbol"), Col("date"), Col("close"), Col("volume"))
    .Sort("volume", true);
 
var dfCombined = lfCombined.Collect();
Console.WriteLine($"High-volume trades: {dfCombined.Shape}");
 
dfCombined.Head(10)
High-volume trades: (14330, 4)
symboldateclosevolume
ISP.MI2023-08-082.338376391539
SAN.MC2021-10-203.36367211467
ISP.MI2023-05-312.1555317362978
ISP.MI2023-03-132.3305311886033
SAN.MC2021-11-033.31306973344

Polars.NET | Lazy GroupBy with aggregation

.GroupBy().Agg() in lazy mode builds the grouping and aggregation into the query plan. Polars can optimize the scan to only read the columns referenced in the grouping key and aggregation expressions. The .Sort() after .Agg() runs on the smaller grouped result, not the full dataset.

Lazy groupby aggregation.

var lfGrouped = LazyFrame.ScanParquet(Path.Combine(DATA, "eurostoxx50_ohlcv.parquet"))
    .GroupBy("symbol")
    .Agg(
        Col("close").Mean().Alias("avg_close"),
        Col("volume").Sum().Alias("total_volume"),
        Col("close").Count().Alias("num_days")
    )
    .Sort("total_volume", true);
 
var dfGrouped = lfGrouped.Collect();
Console.WriteLine($"Grouped result: {dfGrouped.Shape}");
 
dfGrouped.Head(10)
Grouped result: (50, 4)
symbolavg_closetotal_volumenum_days
ISP.MI3.1479872071157045419691321
SAN.MC4.42584763555136419181329
ENEL.MI6.820438304326005619341321
BBVA.MC8.651954101221337731941329
UCG.MI28.45710447183668010991321

Polars.NET | Lazy WithColumns: add computed columns

.WithColumns() adds new expression-based columns to the query plan without materializing the DataFrame. All original columns are preserved alongside the new ones. Expressions passed to .WithColumns() are evaluated lazily — they run only at .Collect() time, after filters have reduced the row count.

WithColumns() demo.

var lfWithCols = LazyFrame.ScanParquet(Path.Combine(DATA, "eurostoxx50_ohlcv.parquet"))
    .Filter(Col("symbol") == Lit("SAP.DE"))
    .WithColumns(
        (Col("high") - Col("low")).Alias("daily_range"),
        (Col("close") - Col("open")).Alias("daily_change")
    )
    .Select(Col("date"), Col("open"), Col("close"), Col("daily_range"), Col("daily_change"));
 
var dfWithCols = lfWithCols.Collect();
Console.WriteLine($"With computed columns: {dfWithCols.Shape}");
 
dfWithCols.Head(5)
With computed columns: (1324, 5)
dateopenclosedaily_rangedaily_change
2021-01-04108.1105.323.72-2.78
2021-01-05104.98105.041.740.06
2021-01-06105.14105.482.660.34
2021-01-07105.58104.521.66-1.06
2021-01-08105.14106.181.681.04

Performance Comparison

We compare eager vs lazy execution on real data to measure the impact of query optimization.

  • Eager: read the entire file into memory, then filter and select.
  • Lazy: scan the file, apply filter + select, then collect — Polars only reads what it needs.

We use Stopwatch for timing and average over multiple iterations to reduce noise.

Benchmark Setup

Polars.NET | Benchmark helper

A Stopwatch-based timing helper that runs warmup iterations to stabilize JIT compilation and OS file caching before measuring. Returns average, min, and max across the measured iterations. Warmup is important: the first run after cold-start will always be slower due to JIT and disk cache effects.

Benchmark helper.

static (double avgMs, double minMs, double maxMs) Benchmark(Action action, int warmup = 2, int iterations = 5)
{
    for (int i = 0; i < warmup; i++)
        action();
 
    var times = new double[iterations];
    var sw = new Stopwatch();
 
    for (int i = 0; i < iterations; i++)
    {
        sw.Restart();
        action();
        sw.Stop();
        times[i] = sw.Elapsed.TotalMilliseconds;
    }
 
    return (times.Average(), times.Min(), times.Max());
}
 
Console.WriteLine("Benchmark helper defined.");
Benchmark helper defined.

Eager vs Lazy Benchmarks

Polars.NET | Eager vs Lazy: Parquet read + filter + select

Benchmarks the same query (filter one exchange, select four columns) on a ~12M-row Parquet file using eager and lazy execution. Eager reads all 12 columns into memory first, then filters. Lazy pushes the filter and column selection into the Parquet scanner, reading only what it needs.

Parquet benchmark.

var benchPath = Path.Combine(DATA, "bench_large.parquet");
Console.WriteLine($"Benchmark file: {benchPath}");
 
var (eagerAvg, eagerMin, eagerMax) = Benchmark(() =>
{
    var df = DataFrame.ReadParquet(benchPath);
    var filtered = df.Filter(Col("exchange") == Lit("XAMS"));
    var selected = filtered.Select(Col("symbol"), Col("date"), Col("price"), Col("size"));
}, warmup: 1, iterations: 3);
 
Console.WriteLine($"EAGER  — avg: {eagerAvg:F1} ms  (min: {eagerMin:F1}, max: {eagerMax:F1})");
 
var (lazyAvg, lazyMin, lazyMax) = Benchmark(() =>
{
    var df = LazyFrame.ScanParquet(benchPath)
        .Filter(Col("exchange") == Lit("XAMS"))
        .Select(Col("symbol"), Col("date"), Col("price"), Col("size"))
        .Collect();
}, warmup: 1, iterations: 3);
 
Console.WriteLine($"LAZY   — avg: {lazyAvg:F1} ms  (min: {lazyMin:F1}, max: {lazyMax:F1})");
 
var speedup = eagerAvg / lazyAvg;
Console.WriteLine($"\nLazy is ~{speedup:F1}x faster than eager on this query.");
Benchmark file: ..\data\bench_large.parquet
EAGER  — avg: 141.5 ms  (min: 99.1, max: 218.3)
LAZY   — avg: 15.8 ms  (min: 15.1, max: 16.2)
 
Lazy is ~9.0x faster than eager on this query.

Polars.NET | Eager vs Lazy: CSV read + filter + select

CSV is a row-based format — unlike Parquet, it cannot skip columns without reading the full line. Projection pushdown has no effect on CSV. LazyFrame.ScanCsv() was not available in early Polars.NET builds; the code falls back to an eager-read + .Lazy() hybrid if ScanCsv throws.

Use Parquet for best lazy performance

CSV must be fully scanned line by line regardless of which columns you request. Lazy CSV (ScanCsv) still applies predicate pushdown to skip non-matching rows, but column pruning does not apply. For data that you query repeatedly, convert to Parquet once and get true projection pushdown.

CSV benchmark.

var csvBenchPath = Path.Combine(DATA, "bench_medium.csv");
Console.WriteLine($"CSV benchmark file: {csvBenchPath} (~2.5M rows)");
 
var (csvEagerAvg, csvEagerMin, csvEagerMax) = Benchmark(() =>
{
    var df = DataFrame.ReadCsv(csvBenchPath);
    var filtered = df.Filter(Col("symbol") == Lit("ASML.AS"));
    var selected = filtered.Select(Col("symbol"), Col("date"), Col("close"), Col("volume"));
}, warmup: 1, iterations: 3);
 
Console.WriteLine($"EAGER CSV  — avg: {csvEagerAvg:F1} ms  (min: {csvEagerMin:F1}, max: {csvEagerMax:F1})");
 
try
{
    var (csvLazyAvg, csvLazyMin, csvLazyMax) = Benchmark(() =>
    {
        var df = LazyFrame.ScanCsv(csvBenchPath)
            .Filter(Col("symbol") == Lit("ASML.AS"))
            .Select(Col("symbol"), Col("date"), Col("close"), Col("volume"))
            .Collect();
    }, warmup: 1, iterations: 3);
 
    Console.WriteLine($"LAZY  CSV  — avg: {csvLazyAvg:F1} ms  (min: {csvLazyMin:F1}, max: {csvLazyMax:F1})");
    Console.WriteLine($"\nLazy CSV speedup: ~{csvEagerAvg / csvLazyAvg:F1}x");
}
catch (Exception ex)
{
    Console.WriteLine($"ScanCsv not available: {ex.GetType().Name}");
    Console.WriteLine("For CSV, use eager ReadCsv + .Lazy() for downstream optimizations.");
 
    var (csvHybridAvg, _, _) = Benchmark(() =>
    {
        var df = DataFrame.ReadCsv(csvBenchPath)
            .Lazy()
            .Filter(Col("symbol") == Lit("ASML.AS"))
            .Select(Col("symbol"), Col("date"), Col("close"), Col("volume"))
            .Collect();
    }, warmup: 1, iterations: 3);
 
    Console.WriteLine($"HYBRID CSV (ReadCsv + .Lazy()) — avg: {csvHybridAvg:F1} ms");
Console.WriteLine("No speedup for CSV reads — the scan cost dominates. Use Parquet for best lazy perf.");
}
CSV benchmark file: ..\data\bench_medium.csv (~2.5M rows)
EAGER CSV  — avg: 103.8 ms  (min: 102.4, max: 105.5)
LAZY  CSV  — avg: 76.8 ms  (min: 75.1, max: 77.7)
 
Lazy CSV speedup: ~1.4x

Projection Impact and Summary

Polars.NET | Projection pushdown impact

Measures the I/O savings from reading 2 columns vs all 12 columns from the same Parquet file. In a columnar format, each column is a separate byte range in the file — selecting fewer columns means less data read from disk, less decompression work, and less memory allocation.

Projection impact.

var projPath = Path.Combine(DATA, "bench_large.parquet");
 
var (allColsAvg, _, _) = Benchmark(() =>
{
    var df = LazyFrame.ScanParquet(projPath).Collect();
}, warmup: 1, iterations: 3);
 
var (twoColsAvg, _, _) = Benchmark(() =>
{
    var df = LazyFrame.ScanParquet(projPath)
        .Select(Col("symbol"), Col("price"))
        .Collect();
}, warmup: 1, iterations: 3);
 
Console.WriteLine($"All 12 columns — avg: {allColsAvg:F1} ms");
Console.WriteLine($"Only 2 columns — avg: {twoColsAvg:F1} ms");
Console.WriteLine($"\nProjection pushdown saves ~{(1 - twoColsAvg / allColsAvg) * 100:F0}% read time.");
All 12 columns — avg: 73.4 ms
Only 2 columns — avg: 17.7 ms
 
Projection pushdown saves ~76% read time.

Polars.NET | Benchmark summary

Assembles the benchmark results into a DataFrame for comparison. The vs_eager column shows the speedup factor relative to the eager Parquet baseline.

Benchmark summary.

var summaryDf = new DataFrame(new Polars.CSharp.Series[]
{
    Polars.CSharp.Series.From("approach", new[] { "Eager Parquet", "Lazy Parquet", "Lazy 2-col Parquet" }),
    Polars.CSharp.Series.From("avg_ms", new[] { eagerAvg, lazyAvg, twoColsAvg }),
    Polars.CSharp.Series.From("vs_eager", new[] { 1.0, eagerAvg / lazyAvg, eagerAvg / twoColsAvg })
});
 
summaryDf
Rendered summary table follows.
approachavg_msvs_eager
Eager Parquet141.53246671
Lazy Parquet15.799833338.957845547
Lazy 2-col Parquet17.74767.974738368

Deedle Note

Deedle has no lazy mode

Deedle loads all data into memory immediately on read. There is no query plan, no predicate pushdown, and no projection pushdown. For large datasets that do not fit comfortably in memory, Polars.NET’s lazy API is the appropriate choice.

Deedle is eager-only. All data is loaded into memory immediately when you read a file. There is no lazy execution mode, no query plan, and no automatic optimization.

For large datasets, Polars.NET’s lazy evaluation with predicate and projection pushdown is significantly faster:

  • Predicate pushdown — filters move into the file reader, so unmatched rows are never materialized.
  • Projection pushdown — unneeded columns are skipped entirely during the Parquet scan.
  • Query optimization — the entire pipeline is rewritten for efficiency before any data moves.

If your workflow fits in memory and you only need basic operations, Deedle works fine. For analytical queries on larger-than-memory data, Polars.NET’s lazy API is the right tool.

Summary

Summary | Lazy API cheat sheet

OperationSyntaxNotes
Scan ParquetLazyFrame.ScanParquet(path)Returns LazyFrame, no data read
Scan CSVLazyFrame.ScanCsv(path)May not exist in 0.4.0; use ReadCsv + .Lazy()
Eager to lazydf.Lazy()Free conversion, no data copy
Collectlf.Collect()Materializes query plan into DataFrame
Explainlf.Explain()Print optimized plan (if available)
Filter (lazy)lf.Filter(Col("x") > Lit(5))Predicate pushdown applies
Select (lazy)lf.Select(Col("a"), Col("b"))Projection pushdown applies
Sort (lazy)lf.Sort("col", descending)Single column per call
GroupBy (lazy)lf.GroupBy("col").Agg(...)Same Agg syntax as eager
WithColumns (lazy)lf.WithColumns(expr.Alias("name"))Add/replace columns in plan

Summary | Key takeaways

  • Always prefer ScanParquet over ReadParquet when you plan to filter or select — predicate and projection pushdown avoid reading unnecessary data.
  • Chain operations lazily — let Polars optimize the full pipeline before execution.
  • Parquet > CSV for lazy — Parquet’s columnar format enables true projection pushdown; CSV must still be fully scanned.
  • Deedle has no lazy mode — every operation is immediate and loads all data.

C# Lazy API and Performance Warnings

Polars.NET DataFrames are immutable — every operation returns a new DataFrame

Forgetting to assign the result of WithColumns(), Filter(), or Sort() silently discards the work. MDA is mutable — column assignment modifies the original.

IfElse in Polars.NET is not When/Then/Otherwise

The C# API uses Col("x").Gt(0).IfElse(trueVal, falseVal) — not When().Then().Otherwise(). Translating from Python literally produces compile errors.

Type mismatches between Polars.NET and MDA are common

Polars.NET uses Arrow types (Int64, Float64, Utf8). MDA uses .NET types (int, double, string). Converting between libraries requires explicit type mapping.

C# Lazy API and Performance Recommendations

  1. Prefer Polars.NET expressions for analytical transforms — the optimizer can fuse and reorder operations.
  2. Use MDA when ML.NET integration is the goal — MDA DataFrame implements IDataView for direct ML.NET handoff.
  3. Validate output schemas after transforms — assert column names and types match expectations.
  4. Prefer Parquet for intermediate data — lossless type preservation between transform steps.

Troubleshooting and failure modes

WithColumns() results are immutable until you assign them

Polars transforms return a new frame, so the original stays unchanged unless you assign the returned value.

Immutable result.

Console.WriteLine("df.WithColumns(...) returns a new DataFrame.");
Console.WriteLine("Assign the result: df = df.WithColumns(...).");
df.WithColumns(...) returns a new DataFrame.
Assign the result: df = df.WithColumns(...).

Cast fails when source values do not fit the target type

Treat cast failures as data-shape problems: clean the source values or branch with IfElse before converting.

Cast failure.

Console.WriteLine("ComputeError on Cast usually means the source values do not fit the target type.");
Console.WriteLine("Clean the data first or branch with `IfElse` before casting.");
ComputeError on Cast usually means the source values do not fit the target type.
Clean the data first or branch with `IfElse` before casting.

MDA column constructors must match the CLR type

Choose the MDA column constructor that matches the CLR type exactly so the column is built with the expected schema.

CLR type mapping.

Console.WriteLine("Use `Int32DataFrameColumn` for `int`, `DoubleDataFrameColumn` for `double`, and `StringDataFrameColumn` for `string`.");
Console.WriteLine("A CLR type mismatch surfaces when the column is constructed.");
Use `Int32DataFrameColumn` for `int`, `DoubleDataFrameColumn` for `double`, and `StringDataFrameColumn` for `string`.
A CLR type mismatch surfaces when the column is constructed.

Graph View

Backlinks