03 - Transforms, Expressions & Chaining - C#

Transforms, Expressions & Chaining - C#

Quote

“If you torture the data long enough, it will confess to anything.”

— Ronald Coase, attributed remark (c. 1960s)

Summary

Covers the core transformation model for C# dataframes by contrasting Polars.NET’s expression-driven, immutable pipeline style with Microsoft.Data.Analysis’ eager typed-column mechanics. The note exists to show how derived columns, conditional logic, casts, window-style calculations, and custom code paths should be expressed when the same business transformation must survive beyond an ad hoc notebook cell.

Column Transforms

• Build and overwrite columns with WithColumns, explicit MDA column construction, arithmetic derivations, percentage-change calculations, and multiple transforms in one logical pass
• Show the practical difference between immutable expression returns in Polars.NET and explicit eager materialization in MDA when creating or replacing analytical fields

Expression System

• Use Col, Lit, and Alias as the primitives for reusable Polars expressions, then layer IfElse, horizontal arithmetic, and string logic on top of them
• Contrast this with MDA, where the same intent is implemented by constructing typed columns directly instead of composing a reusable transform graph

Type Casting

• Cast numeric and date values for downstream arithmetic and parsing, including Cast(DataType.X) in Polars.NET and explicit typed-column replacement in MDA
• Handle categorical encoding and type-shape transitions where CLR-native and Arrow-native representations diverge

Method Chaining & Window Functions

• Compose fluent multi-step pipelines, then apply window functions, rolling aggregates, cumulative calculations, rank logic, and percent-change patterns over financial data
• Emphasize how Polars expressions stay composable across longer chains while MDA requires more explicit step-by-step materialization

Apply / Map / UDF

• Use element-wise UDFs and row-wise logic only where built-in expression patterns stop being sufficient, and understand the cost of dropping out of vectorized dataframe-native operations

Operations and safety

• Warnings: Polars.NET immutability requires reassignment, IfElse is not Python-style When/Then/Otherwise, and Arrow-versus-CLR type mismatches are common across libraries
• Recommendations: 4 practices covering Polars expressions for analytical transforms, MDA for ML.NET-oriented boundaries, schema validation after transforms, and Parquet for type-safe intermediate persistence
• Troubleshooting: 3 failure modes covering unchanged transform results from missed reassignment, ComputeError during casts, and MDA typed-column mismatches

Glossary

WithColumns

• The Polars.NET method for adding or replacing one or more columns through expressions.

• It matters because most non-trivial dataframe transforms in the note start here, and the method defines the immutable “return a new frame” style of Polars work.

No in-place mutation

WithColumns does not mutate the source dataframe. If you do not assign the returned frame, the transformation is effectively discarded.

---

IfElse

• The Polars.NET conditional expression form that chooses between true and false branches based on a boolean expression.

• It matters because conditional logic, classification, binning, and label construction in the note all rely on this API shape.

Not Python when

Translating Python Polars literally will fail here. Polars.NET 0.4.0 expresses this pattern through IfElse, not when().then().otherwise().

---

Cast

• A conversion operation that changes a column from one physical or logical type to another.

• It matters because many transforms depend on moving between integer, float, string, date, or categorical representations before the next calculation is valid.

Dirty values break casts

Casting is not a magic cleanup step. If the input values cannot be represented in the target type, the cast fails and the pipeline needs a guard or pre-cleaning path.

---

Expression

• A composable description of column logic in Polars.NET built from references, literals, operators, and method calls.

• It matters because the note’s central contrast is expression-graph thinking in Polars versus explicit materialized column mechanics in MDA.

Description, not immediate execution

Expressions are reusable transformation definitions. They become concrete only when passed into dataframe operations that evaluate them.

---

Col / Lit / Alias

• The three core expression-building primitives for column references, literal values, and output naming in Polars.NET.

• They matter because nearly every reusable transform in the note is assembled from these primitives before arithmetic, conditionals, or casting are layered on top.

Small primitives, large composability

Once these three pieces are understood, many seemingly different transforms reduce to the same reusable pattern with different operators and target names.

---

Typed column

• An MDA column whose CLR type is fixed explicitly, such as PrimitiveDataFrameColumn<double> or StringDataFrameColumn.

• It matters because Microsoft.Data.Analysis expresses most transforms by building or replacing typed columns directly rather than composing an expression tree.

Type choice is an API contract

In MDA, the target column type is part of the transform design. Choosing the wrong CLR type creates downstream friction immediately.

---

Immutability

• A model where dataframe operations produce new objects instead of modifying the original object in place.

• It matters because Polars.NET uses immutability as a default, and many early mistakes in the note come from forgetting that assignment is part of the transform.

Silent no-op feeling

A missed reassignment often looks like the transform “did nothing,” when the real problem is that the new frame was never captured.

---

Method chaining

• Writing a sequence of dataframe operations as one fluent pipeline where each step feeds the next.

• It matters because the note moves from single-column transforms into multi-step pipelines that are easier to audit when the chain stays readable and intentional.

Chaining exposes model quality

Fluent pipelines stay elegant only when the underlying API composes cleanly. That is one of the clearest practical differences between Polars.NET and MDA in this chapter.

---

Window function

• A calculation that uses neighboring or partition-related rows without collapsing the dataframe to one row per group.

• It matters because rolling metrics, cumulative calculations, rank, and percent-change logic all depend on window-style semantics.

Row context without group collapse

Window logic lets you calculate per-row analytical features while still keeping the original row grain intact.

---

Rolling aggregate

• A moving calculation, such as a rolling mean or sum, evaluated across a sliding window of rows.

• It matters because time-series analytics in the note rely on rolling features rather than only whole-column aggregates.

Ordering is part of correctness

Rolling results are only meaningful if the dataframe is sorted on the intended sequence dimension before the window is applied.

---

Cumulative operation

• A running calculation such as cumulative sum or cumulative count that carries state forward through an ordered sequence.

• It matters because cumulative metrics are a common transformation boundary between raw event rows and usable analytical signals.

Sequence dependence is explicit

Cumulative logic is never purely column-local. The current row depends on every earlier row in the chosen ordering.

---

UDF

• A user-defined function applied when built-in dataframe expressions are insufficient for a particular transformation.

• It matters because the note places UDFs late in the design stack, after vectorized and native expression options have been considered.

Escape hatch, not default

Custom code is flexible, but it is also where composability, optimizer visibility, and much of the dataframe engine’s performance advantage begin to weaken.

---

Suppress CS1701/CS1702 assembly version warnings in .NET Interactive. Run this cell once before any cells that use NuGet packages.

Runs the notebook warning-suppression setup cell. Runs the example cell.

using System.Reflection;
using Microsoft.DotNet.Interactive;
using Microsoft.DotNet.Interactive.CSharp;
 
var csharpKernel = (CSharpKernel)Kernel.Root.FindKernelByName("csharp");
var optionsField = typeof(CSharpKernel).GetField("_scriptOptions",
    BindingFlags.NonPublic | BindingFlags.Instance);
 
var scriptOptions = optionsField.GetValue(csharpKernel);
var withWarningLevel = scriptOptions.GetType().GetMethod("WithWarningLevel");
var newOptions = withWarningLevel.Invoke(scriptOptions, new object[] { 0 });
optionsField.SetValue(csharpKernel, newOptions);

Install NuGet packages and import namespaces. Alias Microsoft.Data.Analysis as MDA so DataFrame continues to refer to Polars.NET inside mixed examples.

Loads the notebook packages, aliases, and HTML formatters used by the rest of the note. Runs the example cell.

#r "nuget: Polars.NET, 0.4.0"
#r "nuget: Polars.NET.Native.win-x64, 0.4.0"
#r "nuget: Microsoft.Data.Analysis, 0.23.0"
 
using System;
using System.IO;
using System.Linq;
using System.Collections.Generic;
using System.Text.RegularExpressions;
using Polars.CSharp;
using static Polars.CSharp.Polars;
using MDA = Microsoft.Data.Analysis;
using Microsoft.DotNet.Interactive.Formatting;
 
Formatter.Register<DataFrame>((df, writer) =>
{
    var html = df.ToHtml();
    html = System.Text.RegularExpressions.Regex.Replace(html, @"(&gt;|>)(.+?)(&lt;|<)", @"$1$2$3");
    html = System.Text.RegularExpressions.Regex.Replace(html, @">""(.+?)""<", @">$1<");
    writer.Write(html);
}, "text/html");
Formatter.Register<Polars.CSharp.Series>((s, writer) =>
    writer.Write($"<pre style='font-size:14px'>{s}</pre>"), "text/html");
 
var DATA = Path.Combine("..", "data");
Console.WriteLine($"Data directory: {Path.GetFullPath(DATA)}");

Data directory: c:\Users\aperi\DEV\LANG\data

Load the primary dataset used throughout this notebook. Both libraries read the same CSV, so the rest of the page focuses on transform style, execution model, and notebook ergonomics rather than data differences.

Loads the shared CSV into both dfP and df so later examples can compare the same rows. Runs the example cell.

var dfP = DataFrame.ReadCsv(Path.Combine(DATA, "eurostoxx50_ohlcv.csv"), tryParseDates: true);
var df = MDA.DataFrame.LoadCsv(Path.Combine(DATA, "eurostoxx50_ohlcv.csv"));
 
display($"Polars: {dfP.Shape}  |  MDA: ({df.Rows.Count}, {df.Columns.Count})");

Polars: (66355, 12)  |  MDA: (66355, 12)

Current API and execution-model check | 2026-04

Polars’ expressions guide and lazy optimization guide document expressions, window functions, folds, categorical data, and optimizer-driven execution as first-class concepts. Microsoft’s DataFrame, PrimitiveDataFrameColumn<T>, and StringDataFrameColumn document an eager typed-column API centered on explicit materialization. In practice, Polars favors declarative transform graphs; MDA favors direct CLR-native column operations.

---

Column Transforms

Adding, modifying, and overwriting columns is the core dataframe workflow. Polars.NET expresses these transforms declaratively through .WithColumns(), while Microsoft.Data.Analysis usually computes typed target columns explicitly and then appends them to a cloned or newly constructed frame.

Polars expressions vs MDA materialization

Polars DataFrames stay close to an expression graph: .WithColumns() returns a new frame and the transform can later participate in lazy optimization. Microsoft.Data.Analysis is eager and explicit: Clone(), Columns.Add(...), and new MDA.DataFrame(...) materialize each added column immediately. That makes debugging straightforward, but it also means the developer owns more of the transform plumbing.

Polars.NET / Microsoft.Data.Analysis | Arithmetic Columns

Polars.NET | Add a computed column with WithColumns

.WithColumns() accepts one or more expressions. Each expression references existing columns via Col(), applies arithmetic or logic, and is named with .Alias(). The result is a new DataFrame with the additional column appended.

Computes a range column as high − low for every row in the 66K-row dataset, then previews the first 5 ABI.BR rows confirming intraday ranges between 0.97 and 2.07. Runs the example cell.

var dfRange = dfP.WithColumns(
    (Col("high") - Col("low")).Alias("range"));
 
dfRange.Select("symbol", "date", "high", "low", "range").Head(5)

symbol	date	high	low	range
ABI.BR	2021-01-04	58.85	56.78	2.07
ABI.BR	2021-01-05	57.98	56.75	1.23
ABI.BR	2021-01-06	58.94	57.39	1.55
ABI.BR	2021-01-07	58.86	57.88	0.98
ABI.BR	2021-01-08	58.4	57.43	0.97

Microsoft.Data.Analysis | Add a computed column by cloning and appending

Microsoft.Data.Analysis does not expose a Polars-style expression builder. The standard pattern is to cast the source columns to typed PrimitiveDataFrameColumn<T> objects, compute the derived column directly, name it with SetName, and append it to a cloned frame.

Computes range = high - low, appends it to a cloned MDA frame, and previews symbol, date, high, low, and range for the first five rows. Runs the example cell.

var highColM = (MDA.PrimitiveDataFrameColumn<float>)df.Columns["high"];
var lowColM = (MDA.PrimitiveDataFrameColumn<float>)df.Columns["low"];
var rangeColM = highColM - lowColM;
rangeColM.SetName("range");
 
var dfRangeM = df.Clone();
dfRangeM.Columns.Add(rangeColM);
 
var selectedM = new MDA.DataFrame(dfRangeM.Columns["symbol"], dfRangeM.Columns["date"], dfRangeM.Columns["high"], dfRangeM.Columns["low"], dfRangeM.Columns["range"]);
selectedM.Head(5)

symbol	date	high	low	range
ABI.BR	2021-01-04 00:00:00Z	58.85	56.78	2.0699997
ABI.BR	2021-01-05 00:00:00Z	57.98	56.75	1.2299995
ABI.BR	2021-01-06 00:00:00Z	58.94	57.39	1.5499992
ABI.BR	2021-01-07 00:00:00Z	58.86	57.88	0.97999954
ABI.BR	2021-01-08 00:00:00Z	58.4	57.43	0.9700012

Polars.NET / Microsoft.Data.Analysis | Column Overwrite via Cast

Polars.NET | Overwrite a column with Cast

When .Alias() matches an existing column name, the new expression replaces that column. .Cast(DataType.X) converts the column’s data type. This is useful for promoting integer columns to float for downstream arithmetic that requires fractional precision.

Casts volume from i64 to f64 by aliasing the expression to the same column name, confirms the dtype change via DataTypeName, and shows the first 3 rows where integer values are preserved as floats. Runs the example cell.

var dfCast = dfP.WithColumns(
    Col("volume").Cast(DataType.Float64)
);
 
display($"Before: {dfP["volume"].DataTypeName}  |  After: {dfCast["volume"].DataTypeName}");
dfCast.Select("symbol", "date", "volume").Head(3)

Before: i64 | After: f64

symbol	date	volume
ABI.BR	2021-01-04	1513937
ABI.BR	2021-01-05	1382722
ABI.BR	2021-01-06	1370204

Microsoft.Data.Analysis | Add a converted numeric column explicitly

MDA type conversion is usually explicit: read the source values from the existing column, create a new typed target column, and populate it row by row. This is more verbose than Polars Cast, but it makes the materialization step obvious and debuggable.

Converts volume into a new volume_f64 column, appends it to a cloned frame, prints the source and target CLR types, and previews the first three rows. Runs the example cell.

var volColM = df.Columns["volume"];
var volDoubleM = new MDA.PrimitiveDataFrameColumn<double>("volume_f64", df.Rows.Count);
 
for(long i = 0; i < df.Rows.Count; i++)
{
    if (volColM[i] != null) volDoubleM[i] = Convert.ToDouble(volColM[i]);
}
 
var dfCastM = df.Clone();
dfCastM.Columns.Add(volDoubleM);
 
display($"Before: {volColM.DataType.Name}  |  After: {volDoubleM.DataType.Name}");
new MDA.DataFrame(dfCastM.Columns["symbol"], dfCastM.Columns["date"], dfCastM.Columns["volume_f64"]).Head(3)

Before: Single  |  After: Double

symbol	date	volume_f64
ABI.BR	2021-01-04 00:00:00Z	1513937
ABI.BR	2021-01-05 00:00:00Z	1382722
ABI.BR	2021-01-06 00:00:00Z	1370204

Polars.NET / Microsoft.Data.Analysis | Percentage Change

Polars.NET | Percentage change with expressions

Compound expressions chain arithmetic operators directly on Col() references. Lit(100.0) injects a scalar constant into the expression tree. The entire expression is evaluated in a single vectorized pass — no intermediate Series objects are allocated.

Computes (close − open) / open * 100 as a single named expression daily_return_pct in one pass, and previews the first 5 ABI.BR rows showing daily returns ranging from −1.62% to +1.40%. Runs the example cell.

var dfPct = dfP.WithColumns(
    ((Col("close") - Col("open")) / Col("open") * Lit(100.0)).Alias("daily_return_pct")
);
 
dfPct.Select("symbol", "date", "open", "close", "daily_return_pct").Head(5)

symbol	date	open	close	daily_return_pct
ABI.BR	2021-01-04	58.15	57.21	-1.616509028
ABI.BR	2021-01-05	56.9	57.18	0.4920913884
ABI.BR	2021-01-06	57.96	58.77	1.397515528
ABI.BR	2021-01-07	58.68	58.4	-0.4771642808
ABI.BR	2021-01-08	58.16	57.86	-0.5158184319

Microsoft.Data.Analysis | Percentage change with typed column arithmetic

Once numeric inputs are typed as PrimitiveDataFrameColumn<float>, MDA supports elementwise arithmetic directly. The transform still materializes eagerly, but the code remains vector-style for straightforward numeric feature engineering.

Computes daily_return_pct = (close - open) / open * 100, appends it to a cloned frame, and previews the first five rows. Runs the example cell.

var openColM = (MDA.PrimitiveDataFrameColumn<float>)df.Columns["open"];
var closeColM = (MDA.PrimitiveDataFrameColumn<float>)df.Columns["close"];
 
var pctColM = (closeColM - openColM) / openColM * 100.0f;
pctColM.SetName("daily_return_pct");
 
var dfPctM = df.Clone();
dfPctM.Columns.Add(pctColM);
 
new MDA.DataFrame(dfPctM.Columns["symbol"], dfPctM.Columns["date"], dfPctM.Columns["open"], dfPctM.Columns["close"], dfPctM.Columns["daily_return_pct"]).Head(5)

symbol	date	open	close	daily_return_pct
ABI.BR	2021-01-04 00:00:00Z	58.15	57.21	-1.6165131
ABI.BR	2021-01-05 00:00:00Z	56.9	57.18	0.49208924
ABI.BR	2021-01-06 00:00:00Z	57.96	58.77	1.3975179
ABI.BR	2021-01-07 00:00:00Z	58.68	58.4	-0.47716218
ABI.BR	2021-01-08 00:00:00Z	58.16	57.86	-0.5158171

Polars.NET / Microsoft.Data.Analysis | Multiple Transforms in One Pass

Polars.NET | Multiple expressions in a single WithColumns call

.WithColumns() accepts multiple comma-separated expressions. Polars evaluates them in a single pass over the data, avoiding repeated scans. This is both more readable and more performant than chaining multiple .WithColumns() calls.

Adds range, midpoint, and daily_return_pct in a single .WithColumns() call with one data scan, and previews 5 ABI.BR rows showing all three derived columns computed simultaneously. Runs the example cell.

var dfMulti = dfP.WithColumns(
    (Col("high") - Col("low")).Alias("range"),
    ((Col("high") + Col("low")) / Lit(2.0)).Alias("midpoint"),
    ((Col("close") - Col("open")) / Col("open") * Lit(100.0)).Alias("daily_return_pct")
);
 
dfMulti.Select("symbol", "date", "range", "midpoint", "daily_return_pct").Head(5)

symbol	date	range	midpoint	daily_return_pct
ABI.BR	2021-01-04	2.07	57.815	-1.616509028
ABI.BR	2021-01-05	1.23	57.365	0.4920913884
ABI.BR	2021-01-06	1.55	58.165	1.397515528
ABI.BR	2021-01-07	0.98	58.37	-0.4771642808
ABI.BR	2021-01-08	0.97	57.915	-0.5158184319

Microsoft.Data.Analysis | Multiple transforms with eager column adds

MDA has no single-call WithColumns equivalent. The practical pattern is to compute each derived column first and then append them to a cloned frame. This keeps the control flow explicit, but each new column is materialized eagerly.

Adds range, midpoint, and daily_return_pct to a cloned frame and previews all three derived columns together. Runs the example cell.

var midColM = (highColM + lowColM) / 2.0f;
midColM.SetName("midpoint");
 
var dfMultiM = df.Clone();
dfMultiM.Columns.Add(rangeColM);
dfMultiM.Columns.Add(midColM);
dfMultiM.Columns.Add(pctColM);
 
new MDA.DataFrame(dfMultiM.Columns["symbol"], dfMultiM.Columns["date"], dfMultiM.Columns["range"], dfMultiM.Columns["midpoint"], dfMultiM.Columns["daily_return_pct"]).Head(5)

symbol	date	range	midpoint	daily_return_pct
ABI.BR	2021-01-04 00:00:00Z	2.0699997	57.815	-1.6165131
ABI.BR	2021-01-05 00:00:00Z	1.2299995	57.364998	0.49208924
ABI.BR	2021-01-06 00:00:00Z	1.5499992	58.165	1.3975179
ABI.BR	2021-01-07 00:00:00Z	0.97999954	58.370003	-0.47716218
ABI.BR	2021-01-08 00:00:00Z	0.9700012	57.915	-0.5158171

Expression System

Polars’ expression engine is the core differentiator in this chapter. Expressions are composable and can later flow into lazy optimization, while Microsoft.Data.Analysis centers the API on eager DataFrame / DataFrameColumn operations and explicit per-column materialization.

Polars expressions vs MDA managed columns

In Polars, Col("x") + Col("y") is an expression object that remains reusable across Select, WithColumns, Filter, window functions, and lazy plans. In Microsoft.Data.Analysis, PrimitiveDataFrameColumn<T> arithmetic executes eagerly and produces a concrete target column immediately. That difference matters most once transformation graphs become long, stateful, or production-bound.

flowchart LR
    A["Col(close)"] --> B["Multiply by Lit(1.10)"]
    B --> C["Alias as close_usd"]
    C --> D[".WithColumns()"]
    D --> E["New DataFrame"]
    style A fill:#292e42,stroke:#565f89,color:#c0caf5
    style B fill:#292e42,stroke:#565f89,color:#c0caf5
    style C fill:#292e42,stroke:#565f89,color:#c0caf5
    style D fill:#24283b,stroke:#7aa2f7,color:#c0caf5
    style E fill:#1a1b26,stroke:#9ece6a,color:#c0caf5

Polars.NET / Microsoft.Data.Analysis | Col, Lit, and Alias

Polars.NET | Col, Lit, and Alias basics

Col("name") references a column by name, Lit(value) injects a scalar constant, and .Alias("name") assigns a name to the resulting expression. These three primitives compose into arbitrarily complex expressions passed to .Select() or .WithColumns().

Selects symbol and close, injects 1.10 as a literal eur_to_usd column, and multiplies close * 1.10 to produce close_usd — showing the first 5 ABI.BR rows with all four columns. Runs the example cell.

var dfExpr = dfP.Select(
    Col("symbol"),
    Col("close"),
    Lit(1.10).Alias("eur_to_usd"),
    (Col("close") * Lit(1.10)).Alias("close_usd")
);
 
dfExpr.Head(5)

symbol	close	eur_to_usd	close_usd
ABI.BR	57.21	1.1	62.931
ABI.BR	57.18	1.1	62.898
ABI.BR	58.77	1.1	64.647
ABI.BR	58.4	1.1	64.24
ABI.BR	57.86	1.1	63.646

Microsoft.Data.Analysis | Explicit constants and typed column arithmetic

MDA does not have Col, Lit, or Alias. To express the same idea, you build the constant column explicitly, apply typed arithmetic on the underlying columns, and materialize the final projection into a new dataframe.

Creates a constant eur_to_usd column, computes close_usd = close * 1.10, and shows the first five rows of the projected result. Runs the example cell.

var eurToUsdColM = new MDA.PrimitiveDataFrameColumn<float>("eur_to_usd", df.Rows.Count);
for (long i = 0; i < df.Rows.Count; i++) eurToUsdColM[i] = 1.10f;
 
var closeUsdColM = closeColM * 1.10f;
closeUsdColM.SetName("close_usd");
 
var dfExprM = new MDA.DataFrame(df.Columns["symbol"], df.Columns["close"], eurToUsdColM, closeUsdColM);
dfExprM.Head(5)

symbol	close	eur_to_usd	close_usd
ABI.BR	57.21	1.1	62.931
ABI.BR	57.18	1.1	62.898003
ABI.BR	58.77	1.1	64.647
ABI.BR	58.4	1.1	64.240005
ABI.BR	57.86	1.1	63.646004

Polars.NET / Microsoft.Data.Analysis | Conditional Expressions

Polars.NET | Conditional column with IfElse

IfElse(condition, true_value, false_value) is the Polars.NET conditional expression. It evaluates the condition per row and returns the corresponding value. Nest IfElse calls for multi-branch logic. The condition, true, and false branches are all expressions — they can reference columns, literals, or further nested expressions.

Builds a return_pct column, then classifies each row as “up”, “down”, or “flat” using two nested IfElse calls on close vs. open, showing 8 ABI.BR rows where 5 of 8 sessions close below the open.

Polars.NET uses IfElse, not When/Then/Otherwise

The Python Polars API uses pl.when().then().otherwise() for conditional logic. Polars.NET 0.4.0 does not expose this API — use IfElse() instead. The summary table at the bottom of this page reflects this difference.

Correct Polars.NET conditional pattern

IfElse(Col("a") > Col("b"), Lit("yes"), Lit("no")).Alias("result")

Runs the conditional-expression example described above. Runs the example cell.

var dailyRet = (Col("close") - Col("open")) / Col("open") * Lit(100.0);
 
var dfCond = dfP.WithColumns(
    dailyRet.Alias("return_pct"),
    IfElse(
        Col("close") > Col("open"),
        Lit("up"),
        IfElse(Col("close") < Col("open"), Lit("down"), Lit("flat"))
    ).Alias("direction")
);
 
dfCond.Select("symbol", "date", "open", "close", "return_pct", "direction").Head(8)

symbol	date	open	close	return_pct	direction
ABI.BR	2021-01-04	58.15	57.21	-1.616509028	down
ABI.BR	2021-01-05	56.9	57.18	0.4920913884	up
ABI.BR	2021-01-06	57.96	58.77	1.397515528	up
ABI.BR	2021-01-07	58.68	58.4	-0.4771642808	down
ABI.BR	2021-01-08	58.16	57.86	-0.5158184319	down

Microsoft.Data.Analysis | Conditional column with an explicit string pass

Conditional logic in MDA is usually an explicit per-row pass that writes the result into a StringDataFrameColumn or a typed boolean column. That is straightforward to debug, but it does not become a reusable expression tree the way it does in Polars.

Builds a direction string column from close versus open, appends both daily_return_pct and direction, and previews the first eight rows. Runs the example cell.

var dirColM = new MDA.StringDataFrameColumn("direction", df.Rows.Count);
for (long i = 0; i < df.Rows.Count; i++)
{
    float c = closeColM[i].GetValueOrDefault();
    float o = openColM[i].GetValueOrDefault();
    dirColM[i] = c > o ? "up" : (c < o ? "down" : "flat");
}
 
var dfCondM = df.Clone();
dfCondM.Columns.Add(pctColM);
dfCondM.Columns.Add(dirColM);
 
new MDA.DataFrame(dfCondM.Columns["symbol"], dfCondM.Columns["date"], dfCondM.Columns["open"], dfCondM.Columns["close"], dfCondM.Columns["daily_return_pct"], dfCondM.Columns["direction"]).Head(8)

symbol	date	open	close	daily_return_pct	direction
ABI.BR	2021-01-04 00:00:00Z	58.15	57.21	-1.6165131	down
ABI.BR	2021-01-05 00:00:00Z	56.9	57.18	0.49208924	up
ABI.BR	2021-01-06 00:00:00Z	57.96	58.77	1.3975179	up
ABI.BR	2021-01-07 00:00:00Z	58.68	58.4	-0.47716218	down
ABI.BR	2021-01-08 00:00:00Z	58.16	57.86	-0.5158171	down

Polars.NET | Nested IfElse for multiple conditions

For multi-tier classification, nest IfElse calls — the false branch of each outer IfElse becomes the next condition. The volume column is cast to Float64 first because Lit() with numeric constants produces float comparisons.

Classifies all 66K rows into “low” / “medium” / “high” / “very_high” volume tiers using three nested IfElse calls, then groups by tier to confirm 27K low, 25K medium, 8.4K very_high, and 5.9K high rows. Runs the example cell.

var volFloat = Col("volume").Cast(DataType.Float64);
 
var dfTier = dfP.WithColumns(
    IfElse(volFloat > Lit(10_000_000.0), Lit("very_high"),
        IfElse(volFloat > Lit(5_000_000.0), Lit("high"),
            IfElse(volFloat > Lit(1_000_000.0), Lit("medium"), Lit("low"))))
    .Alias("vol_tier")
);
 
display(dfTier.Select("symbol", "date", "volume", "vol_tier").Head(8));
 
dfTier.GroupBy("vol_tier").Agg(Col("vol_tier").Count().Alias("count"))

symbol	date	volume	vol_tier
ABI.BR	2021-01-04	1513937	medium
ABI.BR	2021-01-05	1382722	medium
ABI.BR	2021-01-06	1370204	medium
ABI.BR	2021-01-07	1469911	medium
ABI.BR	2021-01-08	1428681	medium

vol_tier	count
medium	24964
low	27061
high	5887
very_high	8443

Microsoft.Data.Analysis | Multi-branch classification with manual thresholds

Multi-branch logic is just an explicit pass over the typed source column. This notebook writes the final label into a string column and then uses ValueCounts() to validate the distribution across the full dataset.

Classifies each row into low, medium, high, or very_high volume tiers, previews the first eight rows, and then counts the tier distribution. Runs the example cell.

var tierColM = new MDA.StringDataFrameColumn("vol_tier", df.Rows.Count);
for (long i = 0; i < df.Rows.Count; i++)
{
    double v = volDoubleM[i].GetValueOrDefault();
    if (v > 10000000) tierColM[i] = "very_high";
    else if (v > 5000000) tierColM[i] = "high";
    else if (v > 1000000) tierColM[i] = "medium";
    else tierColM[i] = "low";
}
 
var dfTierM = df.Clone();
dfTierM.Columns.Add(tierColM);
 
display(new MDA.DataFrame(dfTierM.Columns["symbol"], dfTierM.Columns["date"], dfTierM.Columns["volume"], dfTierM.Columns["vol_tier"]).Head(8));
dfTierM.Columns["vol_tier"].ValueCounts()

symbol	date	volume	vol_tier
ABI.BR	2021-01-04 00:00:00Z	1513937	medium
ABI.BR	2021-01-05 00:00:00Z	1382722	medium
ABI.BR	2021-01-06 00:00:00Z	1370204	medium
ABI.BR	2021-01-07 00:00:00Z	1469911	medium
ABI.BR	2021-01-08 00:00:00Z	1428681	medium

Values	Counts
medium	24964
low	27061
high	5887
very_high	8443

Polars.NET / Microsoft.Data.Analysis | Horizontal Arithmetic

Polars.NET | Horizontal arithmetic across columns

Horizontal operations combine values across multiple columns within each row. Polars.NET 0.4.0 does not expose horizontal_mean, so the OHLC average is computed as manual arithmetic over four Col() references divided by Lit(4.0).

Sums open, high, low, and close in a single expression divided by Lit(4.0) to produce ohlc_avg, and shows 5 ABI.BR rows with per-row OHLC averages between 57.20 and 58.46. Runs the example cell.

var dfHoriz = dfP.WithColumns(
    ((Col("open") + Col("high") + Col("low") + Col("close")) / Lit(4.0)).Alias("ohlc_avg")
);
 
dfHoriz.Select("symbol", "date", "open", "high", "low", "close", "ohlc_avg").Head(5)

symbol	date	open	high	low	close	ohlc_avg
ABI.BR	2021-01-04	58.15	58.85	56.78	57.21	57.7475
ABI.BR	2021-01-05	56.9	57.98	56.75	57.18	57.2025
ABI.BR	2021-01-06	57.96	58.94	57.39	58.77	58.265
ABI.BR	2021-01-07	58.68	58.86	57.88	58.4	58.455
ABI.BR	2021-01-08	58.16	58.4	57.43	57.86	57.9625

Microsoft.Data.Analysis | Horizontal arithmetic with column addition

Horizontal row-wise math is still written as typed column arithmetic. Because there is no horizontal expression namespace, the notebook computes the OHLC average directly from the four typed numeric columns.

Computes ohlc_avg from open, high, low, and close, appends it to a cloned frame, and previews the first five rows. Runs the example cell.

var ohlcAvgM = (openColM + highColM + lowColM + closeColM) / 4.0f;
ohlcAvgM.SetName("ohlc_avg");
 
var dfHorizM = df.Clone();
dfHorizM.Columns.Add(ohlcAvgM);
 
new MDA.DataFrame(dfHorizM.Columns["symbol"], dfHorizM.Columns["date"], dfHorizM.Columns["open"], dfHorizM.Columns["high"], dfHorizM.Columns["low"], dfHorizM.Columns["close"], dfHorizM.Columns["ohlc_avg"]).Head(5)

symbol	date	open	high	low	close	ohlc_avg
ABI.BR	2021-01-04 00:00:00Z	58.15	58.85	56.78	57.21	57.747498
ABI.BR	2021-01-05 00:00:00Z	56.9	57.98	56.75	57.18	57.2025
ABI.BR	2021-01-06 00:00:00Z	57.96	58.94	57.39	58.77	58.265
ABI.BR	2021-01-07 00:00:00Z	58.68	58.86	57.88	58.4	58.455
ABI.BR	2021-01-08 00:00:00Z	58.16	58.4	57.43	57.86	57.962498

Polars.NET / Microsoft.Data.Analysis | String Operations

Polars.NET | Build a direction tag with IfElse

String-valued expressions work the same way as numeric ones — Lit("UP") creates a string constant. This example builds a directional label per row using nested IfElse.

Assigns “UP”, “DOWN”, or “FLAT” to a tag column using two nested IfElse expressions on close vs. open, showing 5 ABI.BR rows where the first row tags as “DOWN” (close 57.21 < open 58.15). Runs the example cell.

var dfLabel = dfP.WithColumns(
    IfElse(
        Col("close") > Col("open"),
        Lit("UP"),
        IfElse(Col("close") < Col("open"), Lit("DOWN"), Lit("FLAT"))
    ).Alias("tag")
);
 
dfLabel.Select("symbol", "date", "close", "open", "tag").Head(5)

symbol	date	close	open	tag
ABI.BR	2021-01-04	57.21	58.15	DOWN
ABI.BR	2021-01-05	57.18	56.9	UP
ABI.BR	2021-01-06	58.77	57.96	UP
ABI.BR	2021-01-07	58.4	58.68	DOWN
ABI.BR	2021-01-08	57.86	58.16	DOWN

Microsoft.Data.Analysis | Build a direction tag with explicit string assignment

MDA string transforms usually mean allocating a StringDataFrameColumn and filling it inside a loop. For short label-generation logic, that is perfectly serviceable and keeps the resulting column strongly associated with the source frame.

Builds a tag column with UP, DOWN, or FLAT based on close versus open and previews the first five rows. Runs the example cell.

var tagColM = new MDA.StringDataFrameColumn("tag", df.Rows.Count);
for (long i = 0; i < df.Rows.Count; i++)
{
    float c = closeColM[i].GetValueOrDefault();
    float o = openColM[i].GetValueOrDefault();
    tagColM[i] = c > o ? "UP" : (c < o ? "DOWN" : "FLAT");
}
 
var dfLabelM = df.Clone();
dfLabelM.Columns.Add(tagColM);
new MDA.DataFrame(dfLabelM.Columns["symbol"], dfLabelM.Columns["date"], dfLabelM.Columns["close"], dfLabelM.Columns["open"], dfLabelM.Columns["tag"]).Head(5)

symbol	date	close	open	tag
ABI.BR	2021-01-04 00:00:00Z	57.21	58.15	DOWN
ABI.BR	2021-01-05 00:00:00Z	57.18	56.9	UP
ABI.BR	2021-01-06 00:00:00Z	58.77	57.96	UP
ABI.BR	2021-01-07 00:00:00Z	58.4	58.68	DOWN
ABI.BR	2021-01-08 00:00:00Z	57.86	58.16	DOWN

Type Casting

Type casting converts column data into the representation your downstream code actually needs. Polars.NET keeps casting inside expressions; Microsoft.Data.Analysis usually builds an explicitly typed target column and populates it value by value or through typed column arithmetic.

Polars.NET / Microsoft.Data.Analysis | Numeric Cast

Polars.NET | Cast a column to Float64 with Col.Cast

.Cast(DataType.Float64) converts the column’s underlying storage type. When aliased to a new name, the original column is preserved alongside the cast version.

Casts id (i64) to a new id_float (f64) column using a different alias, confirms both dtype names with DataTypeName, and shows 3 rows where integer values 21160–21162 are preserved in both columns. Runs the example cell.

var dfCast1 = dfP.WithColumns(
    Col("id").Cast(DataType.Float64).Alias("id_float")
);
 
display($"id dtype: {dfCast1["id"].DataTypeName}  |  id_float dtype: {dfCast1["id_float"].DataTypeName}");
dfCast1.Select("id", "id_float").Head(3)

id dtype: i64 | id_float dtype: f64

id	id_float
21160	21160
21161	21161
21162	21162

Microsoft.Data.Analysis | Convert id to float with a typed target column

MDA exposes each column’s CLR type, but cross-type conversion still usually means constructing a new typed target column and populating it explicitly. This example also checks for column existence first so the notebook fails gracefully if the CSV schema changes.

Checks that id exists, converts it into id_float, prints both data types, and previews the first three rows. Runs the example cell.

MDA.DataFrame resultM = null;
 
if (df.Columns.IndexOf("id") >= 0)
{
    var idColM = df.Columns["id"];
    var idFloatM = new MDA.PrimitiveDataFrameColumn<double>("id_float", df.Rows.Count);
    for (long i = 0; i < df.Rows.Count; i++) if (idColM[i] != null) idFloatM[i] = Convert.ToDouble(idColM[i]);
 
    Console.WriteLine($"id dtype: {idColM.DataType.Name}  |  id_float dtype: {idFloatM.DataType.Name}");
 
    resultM = new MDA.DataFrame(idColM, idFloatM).Head(3);
}
else
{
    Console.WriteLine("'id' column not present in this CSV structure.");
}
 
resultM

id dtype: Single  |  id_float dtype: Double

id	id_float
21160	21160
21161	21161
21162	21162

Polars.NET / Microsoft.Data.Analysis | Date Parsing

Polars.NET | Parse string dates with Str.ToDate

.Str.ToDate(format) parses a string column into a Polars Date type using a strftime format string. Since tryParseDates: true already parsed the date column during CSV load, this example first casts the date back to string to demonstrate the parsing roundtrip.

First casts date back to str to simulate a raw string input, then re-parses it with Str.ToDate("%Y-%m-%d"), confirming the roundtrip restores the date dtype and that 3 sample rows show the same 2021-01-04/05/06 values. Runs the example cell.

var dfDateStr = dfP.WithColumns(
    Col("date").Cast(DataType.String).Alias("date_str")
);
 
var dfDateParsed = dfDateStr.WithColumns(
    Col("date_str").Str.ToDate("%Y-%m-%d").Alias("date_parsed")
);
 
display($"date_str dtype: {dfDateParsed["date_str"].DataTypeName}  |  date_parsed dtype: {dfDateParsed["date_parsed"].DataTypeName}");
dfDateParsed.Select("date_str", "date_parsed").Head(3)

date_str dtype: str | date_parsed dtype: date

date_str	date_parsed
2021-01-04	2021-01-04
2021-01-05	2021-01-05
2021-01-06	2021-01-06

Microsoft.Data.Analysis | Parse string dates into a typed DateTime column

Date parsing is explicit in MDA: allocate a PrimitiveDataFrameColumn<DateTime> and write parsed values only when DateTime.TryParse succeeds. That gives you predictable null behavior when parsing semi-clean strings.

Parses date into a new date_parsed column, prints both data types, and previews the first three rows. Runs the example cell.

var dateStrColM = df.Columns["date"];
var dateParsedColM = new MDA.PrimitiveDataFrameColumn<DateTime>("date_parsed", df.Rows.Count);
 
for (long i = 0; i < df.Rows.Count; i++)
{
    var val = dateStrColM[i]?.ToString();
    if (DateTime.TryParse(val, out var dt)) dateParsedColM[i] = dt;
}
 
display($"date dtype: {dateStrColM.DataType.Name}  |  date_parsed dtype: {dateParsedColM.DataType.Name}");
new MDA.DataFrame(dateStrColM, dateParsedColM).Head(3)

date dtype: DateTime  |  date_parsed dtype: DateTime

date	date_parsed
2021-01-04 00:00:00Z	2021-01-04 00:00:00Z
2021-01-05 00:00:00Z	2021-01-05 00:00:00Z
2021-01-06 00:00:00Z	2021-01-06 00:00:00Z

Polars.NET / Microsoft.Data.Analysis | Categorical Encoding

Polars.NET | Cast to Categorical for memory-efficient string storage

.Cast(DataType.Categorical) dictionary-encodes the column — each unique string is stored once, and the column stores integer codes. This dramatically reduces memory for columns with high repetition (e.g., 66K rows but only 50 unique symbols).

Casts symbol to a cat column aliased as symbol_cat, reports the dtype change from str to cat and 50 unique values, then shows 3 rows confirming that display values remain human-readable strings. Runs the example cell.

var dfCat = dfP.WithColumns(
    Col("symbol").Cast(DataType.Categorical).Alias("symbol_cat")
);
 
display($"symbol dtype: {dfCat["symbol"].DataTypeName}  |  symbol_cat dtype: {dfCat["symbol_cat"].DataTypeName}");
display($"Unique symbols: {dfCat["symbol_cat"].NUnique}");
dfCat.Select("symbol", "symbol_cat").Head(3)

symbol dtype: str | symbol_cat dtype: cat

Unique symbols: 50

symbol	symbol
ABI.BR	ABI.BR
ABI.BR	ABI.BR
ABI.BR	ABI.BR

Microsoft.Data.Analysis | No native categorical type in the dataframe API

Unlike Polars categorical casting, the MDA dataframe API centers on primitive and string columns. A practical notebook pattern is to keep the string column and inspect distinct cardinality with ValueCounts(); if you need encoded features, create numeric codes explicitly at the model-facing layer.

Reports the symbol column type, counts distinct values with ValueCounts(), and previews the first three rows of the original string column.

Where should encoding happen?

Keep business-readable labels in the dataframe while you are exploring or validating the data. If you need stable numeric encoding for a model, build that encoding deliberately in the feature-engineering or ML pipeline stage instead of assuming the dataframe library has a native categorical abstraction.

Runs the distinct-count check against the original MDA string column. Runs the example cell.

var symbolStrColM = (MDA.StringDataFrameColumn)df.Columns["symbol"];
var uniqueCountM = symbolStrColM.ValueCounts().Rows.Count;
 
display($"symbol dtype: {symbolStrColM.DataType.Name}");
display($"Unique symbols: {uniqueCountM}");
new MDA.DataFrame(symbolStrColM).Head(3)

symbol dtype: String

Unique symbols: 50

symbol
ABI.BR
ABI.BR
ABI.BR

Method Chaining & Window Functions

Method chaining composes multiple dataframe operations into a readable pipeline. In Polars.NET, the chain is still expression-oriented and can later move toward lazy execution. In Microsoft.Data.Analysis, the same work is usually expressed as a sequence of named intermediate frames and typed columns.

When should rolling or window logic stay in the dataframe layer?

If the data is already local and the goal is notebook-side feature engineering, either library is acceptable. If the same rolling or window logic becomes a recurring production transform over larger partitions, prefer Polars or move the computation upstream into SQL, Spark, or a streaming engine so you do not own custom loop code as pipeline infrastructure.

Polars.NET / Microsoft.Data.Analysis | Fluent Chaining

Polars.NET | Fluent chain with Filter, WithColumns, Sort, Head, Select

Each method in the chain returns a new DataFrame, allowing .Filter() → .WithColumns() → .Sort() → .Head() → .Select() to read as a single declarative pipeline. The query planner can optimize across the entire chain.

Filters to ASML.AS rows, adds return_pct, sorts ascending by return_pct to surface the 10 worst sessions (led by −16% on 2024-10-15), and projects 5 columns — all in a single fluent expression. Runs the example cell.

var dfChain = dfP
    .Filter(Col("symbol") == Lit("ASML.AS"))
    .WithColumns(
        ((Col("close") - Col("open")) / Col("open") * Lit(100.0)).Alias("return_pct")
    )
    .Sort("return_pct", false)   // ascending: worst sessions first
    .Head(10)
    .Select("symbol", "date", "open", "close", "return_pct");
 
dfChain

symbol	date	open	close	return_pct
ASML.AS	2024-10-15	795.7	668.1	-16.03619455
ASML.AS	2026-01-28	1300	1194.4	-8.123076923
ASML.AS	2024-07-17	946	870.9	-7.938689218
ASML.AS	2025-04-10	623	577.6	-7.287319422
ASML.AS	2022-01-10	667.5	622.3	-6.771535581

Microsoft.Data.Analysis | Equivalent staged pipeline

MDA can express the same pipeline, but each stage is a named intermediate: filter, compute a return column, append it, sort, then trim. That explicitness is useful in debugging-heavy notebook work, but less concise than Polars chaining when transform graphs grow.

Filters to ASML.AS, computes return_pct, sorts ascending by that derived value to surface the same worst sessions as Polars, and projects the final columns. Runs the example cell.

var isAsmlMaskM = (MDA.PrimitiveDataFrameColumn<bool>)((MDA.StringDataFrameColumn)df.Columns["symbol"]).ElementwiseEquals("ASML.AS");
var dfAsmlM = df.Filter(isAsmlMaskM);
 
var dfAsmlChronM = dfAsmlM.OrderBy("date");
var asmlOpenM = (MDA.PrimitiveDataFrameColumn<float>)dfAsmlChronM.Columns["open"];
var asmlCloseM = (MDA.PrimitiveDataFrameColumn<float>)dfAsmlChronM.Columns["close"];
var asmlRetM = (asmlCloseM - asmlOpenM) / asmlOpenM * 100.0f;
asmlRetM.SetName("return_pct");
 
dfAsmlChronM.Columns.Add(asmlRetM);
 
var dfChainM = dfAsmlChronM.OrderBy("return_pct").Head(10);
new MDA.DataFrame(dfChainM.Columns["symbol"], dfChainM.Columns["date"], dfChainM.Columns["open"], dfChainM.Columns["close"], dfChainM.Columns["return_pct"])

symbol | date | open | close | return_pct
ASML.AS | 2024-10-15 | 795.7 | 668.1 | -16.0362
ASML.AS | 2026-01-28 | 1300 | 1194.4 | -8.123075
ASML.AS | 2024-07-17 | 946 | 870.9 | -7.938687
ASML.AS | 2025-04-10 | 623 | 577.6 | -7.287323
ASML.AS | 2022-01-10 | 667.5 | 622.3 | -6.771538
ASML.AS | 2025-07-16 | 669.5 | 625.8 | -6.527261
ASML.AS | 2023-08-24 | 643.5 | 603.6 | -6.20047
ASML.AS | 2022-06-16 | 478.05 | 448.85 | -6.108144
ASML.AS | 2021-09-28 | 708.4 | 665.2 | -6.098251
ASML.AS | 2022-07-05 | 434.2 | 409 | -5.80378

Polars.NET / Microsoft.Data.Analysis | Window Functions

Polars.NET | Group-level mean with Over

.Over("column") is Polars’ window function — it partitions the data by the given column, computes the aggregate within each partition, and broadcasts the result back to every row. This is equivalent to SQL’s AVG(close) OVER (PARTITION BY symbol).

Computes the mean close per symbol across all 50 partitions simultaneously and broadcasts it back to every row, so all 8 ABI.BR rows show the same mean_close_by_symbol of ~54.86. Runs the example cell.

var dfOver = dfP.WithColumns(
    Col("close").Mean().Over("symbol").Alias("mean_close_by_symbol")
);
 
dfOver.Select("symbol", "date", "close", "mean_close_by_symbol").Head(8)

symbol	date	close	mean_close_by_symbol
ABI.BR	2021-01-04	57.21	54.86423366
ABI.BR	2021-01-05	57.18	54.86423366
ABI.BR	2021-01-06	58.77	54.86423366
ABI.BR	2021-01-07	58.4	54.86423366
ABI.BR	2021-01-08	57.86	54.86423366

Microsoft.Data.Analysis | Group-level mean via dictionary broadcast

There is no Over-style window expression in MDA. The equivalent pattern is to compute group aggregates in dictionaries and then broadcast the result back into a new column. This works well for modest in-memory frames but gets verbose for richer window logic.

Computes the mean close per symbol, broadcasts it back into mean_close_by_symbol, and previews the first eight rows. Runs the example cell.

var symbolsM = (MDA.StringDataFrameColumn)df.Columns["symbol"];
var sumsM = new Dictionary<string, double>();
var countsM = new Dictionary<string, int>();
 
for (long i = 0; i < df.Rows.Count; i++)
{
    var s = symbolsM[i];
    var c = closeColM[i];
    if (s != null && c.HasValue)
    {
        if (!sumsM.ContainsKey(s)) { sumsM[s] = 0; countsM[s] = 0; }
        sumsM[s] += c.Value;
        countsM[s]++;
    }
}
 
var meanColM = new MDA.PrimitiveDataFrameColumn<float>("mean_close_by_symbol", df.Rows.Count);
for (long i = 0; i < df.Rows.Count; i++)
{
    var s = symbolsM[i];
    if (s != null && countsM.ContainsKey(s)) meanColM[i] = (float)(sumsM[s] / countsM[s]);
}
 
var dfOverM = df.Clone();
dfOverM.Columns.Add(meanColM);
new MDA.DataFrame(dfOverM.Columns["symbol"], dfOverM.Columns["date"], dfOverM.Columns["close"], dfOverM.Columns["mean_close_by_symbol"]).Head(8)

symbol	date	close	mean_close_by_symbol
ABI.BR	2021-01-04 00:00:00Z	57.21	54.864235
ABI.BR	2021-01-05 00:00:00Z	57.18	54.864235
ABI.BR	2021-01-06 00:00:00Z	58.77	54.864235
ABI.BR	2021-01-07 00:00:00Z	58.4	54.864235
ABI.BR	2021-01-08 00:00:00Z	57.86	54.864235

Polars.NET / Microsoft.Data.Analysis | Rolling Aggregates

Polars.NET | Rolling mean with RollingMean

.RollingMean("20i") computes a rolling average over a window of 20 rows. The "20i" syntax specifies an integer-indexed window (20 rows). Polars fills partial windows at the start of the series with the available data, so the first row’s rolling mean equals the first value itself.

Filters and sorts ASML.AS chronologically, then computes a 20-row rolling mean on close — showing partial windows filling from row 1 (mean=406.25) through row 5 (mean=407.19) up to row 25 where the full 20-row window first applies. Runs the example cell.

var dfAsml = dfP.Filter(Col("symbol") == Lit("ASML.AS")).Sort("date", false);
 
var dfRoll = dfAsml.WithColumns(
    Col("close").RollingMean("20i").Alias("close_ma20")
);
 
dfRoll.Select("symbol", "date", "close", "close_ma20").Head(25)

symbol	date	close	close_ma20
ASML.AS	2021-01-04	406.25	406.25
ASML.AS	2021-01-05	406.9	406.575
ASML.AS	2021-01-06	402.85	405.3333333
ASML.AS	2021-01-07	403.9	404.975
ASML.AS	2021-01-08	416.05	407.19

Microsoft.Data.Analysis | Rolling mean with an explicit sliding loop

Rolling logic in MDA is stateful code over the ordered frame. The example sorts ASML rows and then computes a 20-row moving mean manually. This gives full control over window semantics, but the developer owns every edge-case detail.

Builds a 20-row rolling mean over ASML closes in chronological order and previews the first ten rows. Runs the example cell.

var dfRollBaseChronM = dfAsmlM.OrderBy("date").Clone();
var rollCloseColM = (MDA.PrimitiveDataFrameColumn<float>)dfRollBaseChronM.Columns["close"];
var ma20ColM = new MDA.PrimitiveDataFrameColumn<float>("close_ma20", dfRollBaseChronM.Rows.Count);
 
for (long i = 0; i < dfRollBaseChronM.Rows.Count; i++)
{
    float sum = 0;
    int count = 0;
    for (long j = 0; j < 20 && (i - j) >= 0; j++)
    {
        var val = rollCloseColM[i - j];
        if (val.HasValue) { sum += val.Value; count++; }
    }
    if (count > 0) ma20ColM[i] = sum / count;
}
 
dfRollBaseChronM.Columns.Add(ma20ColM);
new MDA.DataFrame(dfRollBaseChronM.Columns["symbol"], dfRollBaseChronM.Columns["date"], dfRollBaseChronM.Columns["close"], dfRollBaseChronM.Columns["close_ma20"]).Head(10)

symbol | date | close | close_ma20
ASML.AS | 2021-01-04 | 406.25 | 406.25
ASML.AS | 2021-01-05 | 406.9 | 406.575
ASML.AS | 2021-01-06 | 402.85 | 405.3333
ASML.AS | 2021-01-07 | 403.9 | 404.975
ASML.AS | 2021-01-08 | 416.05 | 407.19
ASML.AS | 2021-01-11 | 414.9 | 408.475
ASML.AS | 2021-01-12 | 418.95 | 409.9714
ASML.AS | 2021-01-13 | 422.45 | 411.5312
ASML.AS | 2021-01-14 | 447.35 | 415.5111
ASML.AS | 2021-01-15 | 435.85 | 417.545

Polars.NET / Microsoft.Data.Analysis | Cumulative Operations

Polars.NET | Cumulative sum with CumSum

.CumSum() computes the running total of a column. Combined with .Filter() and .Sort(), this builds a cumulative volume curve for a single symbol. The operation is vectorized and runs in a single pass.

Filters and sorts ASML.AS by date, applies .CumSum() to volume, building a running total that grows from 789K on day 1 to 9.05M by day 10. Runs the example cell.

var dfAsmlCum = dfP
    .Filter(Col("symbol") == Lit("ASML.AS"))
    .Sort("date", false)
    .WithColumns(
        Col("volume").CumSum().Alias("cum_volume")
    );
 
dfAsmlCum.Select("symbol", "date", "volume", "cum_volume").Head(10)

symbol	date	volume	cum_volume
ASML.AS	2021-01-04	789502	789502
ASML.AS	2021-01-05	798787	1588289
ASML.AS	2021-01-06	875711	2464000
ASML.AS	2021-01-07	874780	3338780
ASML.AS	2021-01-08	975243	4314023

Microsoft.Data.Analysis | Cumulative volume with a running accumulator

Cumulative transforms are another explicit-state pattern in MDA: hold the running accumulator in a scalar, write each step into a typed target column, and append the result to the working frame.

Accumulates ASML trading volume into cum_volume and previews the first ten rows. Runs the example cell.

var dfCumBaseM = dfAsmlM.OrderBy("date").Clone();
var cumVolColM = new MDA.PrimitiveDataFrameColumn<double>("cum_volume", dfCumBaseM.Rows.Count);
var rollVolColM = dfCumBaseM.Columns["volume"];
double currentCumM = 0;
 
for (long i = 0; i < dfCumBaseM.Rows.Count; i++)
{
    double v = Convert.ToDouble(rollVolColM[i] ?? 0.0);
    currentCumM += v;
    cumVolColM[i] = currentCumM;
}
 
dfCumBaseM.Columns.Add(cumVolColM);
new MDA.DataFrame(dfCumBaseM.Columns["symbol"], dfCumBaseM.Columns["date"], dfCumBaseM.Columns["volume"], dfCumBaseM.Columns["cum_volume"]).Head(10)

symbol | date | volume | cum_volume
ASML.AS | 2021-01-04 | 789502 | 789502
ASML.AS | 2021-01-05 | 798787 | 1588289
ASML.AS | 2021-01-06 | 875711 | 2464000
ASML.AS | 2021-01-07 | 874780 | 3338780
ASML.AS | 2021-01-08 | 975243 | 4314023
ASML.AS | 2021-01-11 | 717929 | 5031952
ASML.AS | 2021-01-12 | 787472 | 5819424
ASML.AS | 2021-01-13 | 669646 | 6489070
ASML.AS | 2021-01-14 | 1272594 | 7761664
ASML.AS | 2021-01-15 | 1291058 | 9052722

Polars.NET / Microsoft.Data.Analysis | Rank

Polars.NET | Rank with Col.Rank expression

.Rank() assigns a rank to each value. By default, Polars uses the “average” method for ties (e.g., two values tied for rank 11 both receive 11.5). The ranking is computed within the filtered/sorted context — here, close prices for a single symbol.

Filters and sorts ASML.AS by date, ranks all close values across the full history using average-tie handling, and shows 10 rows — confirming rank 6 on 2021-01-04 (close=406.25) as the 6th-lowest ASML close price and rank 11.5 for a tied pair. Runs the example cell.

var dfRank = dfP
    .Filter(Col("symbol") == Lit("ASML.AS"))
    .Sort("date", false)
    .WithColumns(
        Col("close").Rank().Alias("close_rank")
    );
 
dfRank.Select("symbol", "date", "close", "close_rank").Head(10)

symbol	date	close	close_rank
ASML.AS	2021-01-04	406.25	6
ASML.AS	2021-01-05	406.9	7
ASML.AS	2021-01-06	402.85	3
ASML.AS	2021-01-07	403.9	5
ASML.AS	2021-01-08	416.05	13

Microsoft.Data.Analysis | Rank values by sorting indexed observations

MDA has no built-in rank expression, so ranking is a two-step algorithm: gather the non-null values with their row positions, sort them, and then write the rank back into a new numeric column. To keep the comparison aligned with the Polars example, this version assigns average ranks across ties.

Ranks ASML close prices by value with average-tie handling and previews the first ten chronological rows. Runs the example cell.

var dfRankBaseM = dfAsmlM.OrderBy("date").Clone();
var rankCloseColM = (MDA.PrimitiveDataFrameColumn<float>)dfRankBaseM.Columns["close"];
var rankColM = new MDA.PrimitiveDataFrameColumn<double>("close_rank", dfRankBaseM.Rows.Count);
var indexedClosesM = new List<(long Index, float Value)>();
 
for (long i = 0; i < dfRankBaseM.Rows.Count; i++)
{
    if (rankCloseColM[i].HasValue) indexedClosesM.Add((i, rankCloseColM[i].Value));
}
 
var sortedClosesM = indexedClosesM.OrderBy(x => x.Value).ThenBy(x => x.Index).ToList();
for (int pos = 0; pos < sortedClosesM.Count;)
{
    int groupEnd = pos;
    while (groupEnd + 1 < sortedClosesM.Count && sortedClosesM[groupEnd + 1].Value == sortedClosesM[pos].Value)
    {
        groupEnd++;
    }
 
    double avgRank = ((pos + 1) + (groupEnd + 1)) / 2.0;
    for (int i = pos; i <= groupEnd; i++)
    {
        rankColM[sortedClosesM[i].Index] = avgRank;
    }
 
    pos = groupEnd + 1;
}
 
dfRankBaseM.Columns.Add(rankColM);
new MDA.DataFrame(dfRankBaseM.Columns["symbol"], dfRankBaseM.Columns["date"], dfRankBaseM.Columns["close"], dfRankBaseM.Columns["close_rank"]).Head(10)

symbol | date | close | close_rank
ASML.AS | 2021-01-04 | 406.25 | 6
ASML.AS | 2021-01-05 | 406.9 | 7
ASML.AS | 2021-01-06 | 402.85 | 3
ASML.AS | 2021-01-07 | 403.9 | 5
ASML.AS | 2021-01-08 | 416.05 | 13
ASML.AS | 2021-01-11 | 414.9 | 11.5
ASML.AS | 2021-01-12 | 418.95 | 14
ASML.AS | 2021-01-13 | 422.45 | 16
ASML.AS | 2021-01-14 | 447.35 | 44
ASML.AS | 2021-01-15 | 435.85 | 25

Polars.NET / Microsoft.Data.Analysis | Percent Change

Polars.NET | Day-over-day percent change with PctChange

.PctChange(n) computes (current - previous) / previous with a configurable lag. The first row returns null because there is no prior value. This is one of the most common operations in financial time series analysis.

Filters and sorts ASML.AS chronologically, computes .PctChange(1) on close, producing null on 2021-01-04 and fractional daily returns thereafter (e.g., +0.16% on 2021-01-05, then −0.995% on 2021-01-06). Runs the example cell.

var dfPctChg = dfP
    .Filter(Col("symbol") == Lit("ASML.AS"))
    .Sort("date", false)
    .WithColumns(
        Col("close").PctChange(1).Alias("close_pct_change")
    );
 
dfPctChg.Select("symbol", "date", "close", "close_pct_change").Head(10)

symbol	date	close	close_pct_change
ASML.AS	2021-01-04	406.25	null
ASML.AS	2021-01-05	406.9	0.0016
ASML.AS	2021-01-06	402.85	-0.00995330548
ASML.AS	2021-01-07	403.9	0.002606429192
ASML.AS	2021-01-08	416.05	0.03008170339

Microsoft.Data.Analysis | Day-over-day percent change with explicit lag logic

Lag-based transforms are explicit loops in MDA. The current row and prior row are read from the typed price column, and the result is written only when both values are present and the prior value is nonzero.

Computes day-over-day percent change for ASML closes in chronological order and previews the first ten rows. Runs the example cell.

var dfPctBaseM = dfAsmlM.OrderBy("date").Clone();
var pctCloseColM = (MDA.PrimitiveDataFrameColumn<float>)dfPctBaseM.Columns["close"];
var pctChangeColM = new MDA.PrimitiveDataFrameColumn<float>("close_pct_change", dfPctBaseM.Rows.Count);
 
for (long i = 1; i < dfPctBaseM.Rows.Count; i++)
{
    var curr = pctCloseColM[i];
    var prev = pctCloseColM[i - 1];
    if (curr.HasValue && prev.HasValue && prev.Value != 0)
    {
        pctChangeColM[i] = (curr.Value - prev.Value) / prev.Value;
    }
}
 
dfPctBaseM.Columns.Add(pctChangeColM);
new MDA.DataFrame(dfPctBaseM.Columns["symbol"], dfPctBaseM.Columns["date"], dfPctBaseM.Columns["close"], dfPctBaseM.Columns["close_pct_change"]).Head(10)

symbol | date | close | close_pct_change
ASML.AS | 2021-01-04 | 406.25 | <null>
ASML.AS | 2021-01-05 | 406.9 | 0.00159999
ASML.AS | 2021-01-06 | 402.85 | -0.00995328
ASML.AS | 2021-01-07 | 403.9 | 0.0026064
ASML.AS | 2021-01-08 | 416.05 | 0.03008169
ASML.AS | 2021-01-11 | 414.9 | -0.00276408
ASML.AS | 2021-01-12 | 418.95 | 0.00976143
ASML.AS | 2021-01-13 | 422.45 | 0.00835422
ASML.AS | 2021-01-14 | 447.35 | 0.05894187
ASML.AS | 2021-01-15 | 435.85 | -0.02570694

Apply / Map / UDF

User-defined functions (UDFs) apply custom logic element-wise or row-wise. Polars encourages native expressions over UDF-style code because expressions stay vectorized and optimizable. Microsoft.Data.Analysis is more comfortable with explicit CLR loops, but those loops are still eager notebook code rather than reusable query semantics.

Prefer built-in column operations over manual loops

In Polars, expression-based transforms keep the work vectorized and compatible with lazy optimization. In Microsoft.Data.Analysis, typed column arithmetic is still preferable to per-row loops whenever possible because it keeps the code shorter and reduces custom state handling.

Polars.NET / Microsoft.Data.Analysis | Element-wise UDF

Polars.NET | Element-wise UDF via extract-transform-add

MapElements is not available in Polars.NET 0.4.0. The workaround extracts the column to a C# array with .ToArray<T>(), applies a LINQ .Select() transform, wraps the result as a Polars.CSharp.Series, and stacks it onto the DataFrame with .HStack().

Extracts ASML.AS close as a double[], applies Math.Log() element-wise via LINQ, wraps the result as a named Series, and attaches it with .HStack() — showing the first 8 rows with natural log prices between 6.007 and 6.046.

MapElements not available in Polars.NET 0.4.0

The Python Polars map_elements() function has no direct equivalent in the .NET bindings at version 0.4.0. Use the extract-transform-add pattern shown below.

Preferred workaround

Keep the transformation in Polars expressions whenever possible. Use extract-transform-add only for CLR-specific logic that genuinely cannot be expressed with the current Polars.NET API surface.

Runs the extract-transform-add workaround described above. Runs the example cell.

var dfAsmlU = dfP
    .Filter(Col("symbol") == Lit("ASML.AS"))
    .Sort("date");
 
var closeArr = dfAsmlU.Column("close").ToArray<double>();
var logArr = closeArr.Select(v => Math.Log(v)).ToArray();
var logSeries = Polars.CSharp.Series.From("log_close", logArr);
 
var dfUdf = dfAsmlU.HStack(logSeries);
dfUdf.Select("symbol", "date", "close", "log_close").Head(8)

symbol	date	close	log_close
ASML.AS	2021-01-04	406.25	6.006968734
ASML.AS	2021-01-05	406.9	6.008567455
ASML.AS	2021-01-06	402.85	5.998564284
ASML.AS	2021-01-07	403.9	6.001167323
ASML.AS	2021-01-08	416.05	6.030805445

Microsoft.Data.Analysis | Element-wise transform with an explicit target column

In MDA, custom elementwise transforms are usually direct loops over a typed source column. That is mechanically simple and fits CLR-native math well, but it is still eager, notebook-local materialization rather than an optimizable expression.

Applies Math.Log() to ASML close prices, appends log_close, and previews the first eight rows. Runs the example cell.

var dfAsmlLogM = dfAsmlM.OrderBy("date");
var asmlLogCloseColM = (MDA.PrimitiveDataFrameColumn<float>)dfAsmlLogM.Columns["close"];
var logColM = new MDA.PrimitiveDataFrameColumn<double>("log_close", dfAsmlLogM.Rows.Count);
 
for (long i = 0; i < dfAsmlLogM.Rows.Count; i++)
{
    if (asmlLogCloseColM[i].HasValue) logColM[i] = Math.Log(asmlLogCloseColM[i].Value);
}
 
dfAsmlLogM.Columns.Add(logColM);
new MDA.DataFrame(dfAsmlLogM.Columns["symbol"], dfAsmlLogM.Columns["date"], dfAsmlLogM.Columns["close"], dfAsmlLogM.Columns["log_close"]).Head(8)

symbol	date	close	log_close
ASML.AS	2021-01-04 00:00:00Z	406.25	6.006968733643947
ASML.AS	2021-01-05 00:00:00Z	406.9	6.008567440007606
ASML.AS	2021-01-06 00:00:00Z	402.85	5.998564299374044
ASML.AS	2021-01-07 00:00:00Z	403.9	6.001167307457915
ASML.AS	2021-01-08 00:00:00Z	416.05	6.03080541600614

Polars.NET / Microsoft.Data.Analysis | Row-wise Logic

Polars.NET | Row-wise logic with expressions (preferred over UDFs)

Multi-column row-wise conditions are best expressed with Polars’ expression combinators using & (AND) and | (OR). This keeps the operation vectorized and optimizable. The volume column is cast to Float64 for comparison with the Lit() constant.

Flags rows where close > open AND volume > 2,000,000 as bullish_high_vol, finding 13,931 such rows across all 50 symbols — previewing 8 ABI.BR rows which are all false due to low volume. Runs the example cell.

var dfRowWise = dfP.WithColumns(
    IfElse(
        (Col("close") > Col("open")) & (Col("volume").Cast(DataType.Float64) > Lit(2_000_000.0)),
        Lit(true),
        Lit(false)
    ).Alias("bullish_high_vol")
);
 
display($"Bullish high-vol rows: {dfRowWise.Filter(Col("bullish_high_vol") == Lit(true)).Shape}");
dfRowWise.Select("symbol", "date", "close", "open", "volume", "bullish_high_vol").Head(8)

Bullish high-vol rows: (13931, 13)

symbol	date	close	open	volume	bullish_high_vol
ABI.BR	2021-01-04	57.21	58.15	1513937	false
ABI.BR	2021-01-05	57.18	56.9	1382722	false
ABI.BR	2021-01-06	58.77	57.96	1370204	false
ABI.BR	2021-01-07	58.4	58.68	1469911	false
ABI.BR	2021-01-08	57.86	58.16	1428681	false

Microsoft.Data.Analysis | Row-wise boolean flag with an explicit bool column

Multi-column row logic is expressed by reading the relevant typed columns together and writing the boolean outcome into a PrimitiveDataFrameColumn<bool>. This is the clearest MDA pattern when the condition cannot be reduced to a simpler precomputed numeric transform.

Flags rows where close > open and volume > 2,000,000, prints the filtered shape, and previews the first eight rows. Runs the example cell.

var bullishVolColM = new MDA.PrimitiveDataFrameColumn<bool>("bullish_high_vol", df.Rows.Count);
 
for (long i = 0; i < df.Rows.Count; i++)
{
    var c = closeColM[i];
    var o = openColM[i];
    var v = volDoubleM[i];
 
    bullishVolColM[i] = c.HasValue && o.HasValue && v.HasValue
                        && (c.Value > o.Value)
                        && (v.Value > 2_000_000.0);
}
 
var dfRowWiseM = df.Clone();
dfRowWiseM.Columns.Add(bullishVolColM);
 
var filteredBullishM = dfRowWiseM.Filter(bullishVolColM);
display($"Bullish high-vol rows: ({filteredBullishM.Rows.Count}, {filteredBullishM.Columns.Count})");
new MDA.DataFrame(dfRowWiseM.Columns["symbol"], dfRowWiseM.Columns["date"], dfRowWiseM.Columns["close"], dfRowWiseM.Columns["open"], dfRowWiseM.Columns["volume"], dfRowWiseM.Columns["bullish_high_vol"]).Head(8)

Bullish high-vol rows: (13931, 13)

symbol	date	close	open	volume	bullish_high_vol
ABI.BR	2021-01-04 00:00:00Z	57.21	58.15	1513937	False
ABI.BR	2021-01-05 00:00:00Z	57.18	56.9	1382722	False
ABI.BR	2021-01-06 00:00:00Z	58.77	57.96	1370204	False
ABI.BR	2021-01-07 00:00:00Z	58.4	58.68	1469911	False
ABI.BR	2021-01-08 00:00:00Z	57.86	58.16	1428681	False

Summary

This chapter is where the libraries diverge most clearly. Polars.NET gives you a declarative transformation DSL that scales from notebook experimentation into lazy analytical pipelines. Microsoft.Data.Analysis gives you explicit typed-column mechanics that fit .NET-centric preprocessing, ML.NET handoff, and situations where you want total control over every materialized step.

API Comparison

Operation	Polars.NET	Microsoft.Data.Analysis
Add column	df.WithColumns(expr.Alias("name"))	Build a typed column and append with Columns.Add(...) or new MDA.DataFrame(...)
Multiple transforms	Many expressions in one .WithColumns(...)	Compute each target column explicitly and append eagerly
Literal / scalar injection	Lit(value)	Build a constant target column explicitly
Conditional logic	IfElse(cond, x, y)	Manual pass into StringDataFrameColumn or PrimitiveDataFrameColumn<bool>
Horizontal arithmetic	Expression arithmetic or folds	Typed column arithmetic across the source columns
Numeric cast	Col("c").Cast(DataType.X)	Construct a new typed target column and populate it
Date parsing	Col("d").Str.ToDate(fmt)	Parse into PrimitiveDataFrameColumn<DateTime>
Categorical encoding	Col("c").Cast(DataType.Categorical)	No native categorical type in the dataframe API
Fluent chaining	.Filter().WithColumns().Sort().Head()	Named eager stages and intermediate frames
Window / broadcast	Col("c").Mean().Over("g")	Aggregate into dictionaries and broadcast back manually
Rolling mean	Col("c").RollingMean(n)	Explicit sliding loop over the ordered frame
Cumulative sum	Col("c").CumSum()	Running accumulator into a target column
Rank	Col("c").Rank()	Sort indexed observations and write ranks back
Pct change	Col("c").PctChange(n)	Manual prior-row loop
Element UDF	Extract / transform / HStack workaround in 0.4.0	Explicit loop into a target column
Row-wise logic	Prefer expressions with IfElse, &, and `	`

Decision Criteria

Applied architecture choices should stay reversible. Treat WithColumns(), Over(), RollingMean(), and PctChange() as signals that the transform is drifting toward an analytical pipeline rather than notebook-local preprocessing.

Prefer WithColumns() for analytical pipelines

If the work is mostly columnar arithmetic, conditional branching, windows, or repeated rolling features, Polars.NET keeps the logic declarative and easier to migrate beyond a notebook.

Summarizes the operations that most strongly favor the Polars expression engine. Runs the example cell.

var analyticalOps = new[] { "WithColumns", "Over", "RollingMean", "PctChange", "CumSum" };
Console.WriteLine(string.Join(", ", analyticalOps));

WithColumns, Over, RollingMean, PctChange, CumSum

Prefer DataFrame when the boundary is IDataView

If the notebook is preparing features immediately before ML.NET, Microsoft.Data.Analysis stays practical because DataFrame already fits the .NET-native handoff model.

Lists the .NET-specific boundaries that justify staying in MDA. Runs the example cell.

var mdaBoundaries = new[] { "IDataView", "CLR business rules", "explicit typed columns" };
Console.WriteLine(string.Join(" | ", mdaBoundaries));

IDataView | CLR business rules | explicit typed columns

Push repeated transforms upstream when possible

If the same transform could execute earlier in SQL, DuckDB, Spark, or a lakehouse engine, move it there instead of maintaining notebook-local loops and state handling.

Prints the preferred upstream execution boundaries for repeatable transforms. Runs the example cell.

var upstreamTargets = new[] { "SQL", "DuckDB", "Spark", "lakehouse" };
Console.WriteLine($"Upstream first: {string.Join(", ", upstreamTargets)}");

Upstream first: SQL, DuckDB, Spark, lakehouse

Operational Risks

These are the failure modes that most often make a dataframe transform look correct in prose while doing the wrong thing in code.

Reassign immutable Polars results

WithColumns(), Filter(), and Sort() return a new frame. If you do not capture the result, the transform is discarded even though the expression itself is valid.

Prints the safe reassignment pattern for immutable Polars transforms. Runs the example cell.

var safePattern = "dfP = dfP.WithColumns(...)";
Console.WriteLine(safePattern);

dfP = dfP.WithColumns(...)

Keep IfElse() syntax C#-native

IfElse() is the supported conditional shape in this note. Translating Python when().then().otherwise() examples literally will produce the wrong API for the .NET bindings discussed here.

Prints the supported Polars.NET conditional pattern. Runs the example cell.

var ifElsePattern = "IfElse(Col(\"a\") > Col(\"b\"), Lit(\"yes\"), Lit(\"no\"))";
Console.WriteLine(ifElsePattern);

IfElse(Col("a") > Col("b"), Lit("yes"), Lit("no"))

Map Arrow and CLR types deliberately

Cast() does not erase the difference between Arrow-native dtypes and CLR-native column types. Cross-library work stays safer when you decide the target type explicitly before the next transform.

Prints the type families that need deliberate mapping between libraries. Runs the example cell.

var typePairs = new[] { "Int64 -> long", "Float64 -> double", "Utf8 -> string" };
Console.WriteLine(string.Join(" | ", typePairs));

Int64 -> long | Float64 -> double | Utf8 -> string

C# Transforms, Expressions and Chaining Troubleshooting

Use these checks when the code compiles but the resulting frame or schema does not match the intended transform.

Result unchanged after WithColumns()

If a Polars transform appears unchanged, inspect whether the returned frame from WithColumns() or Filter() was assigned back into a working variable such as dfP.

Prints the first check for a no-op-looking Polars transform. Runs the example cell.

Console.WriteLine("Check reassignment before debugging expression logic.");

Check reassignment before debugging expression logic.

Cast() fails on dirty values

If Cast() raises a conversion error, guard the bad values first with IfElse() or clean the string/numeric source before attempting the type transition.

Prints the first remediation step for cast failures. Runs the example cell.

Console.WriteLine("Clean or guard invalid values before Cast().");

Clean or guard invalid values before Cast().

MDA schema mismatch after manual column construction

If an MDA column does not behave correctly downstream, verify that the chosen PrimitiveDataFrameColumn<T> or StringDataFrameColumn matches the CLR type you actually intend to materialize.

Prints the MDA type-check reminder for manual column construction. Runs the example cell.

Console.WriteLine("Match PrimitiveDataFrameColumn<T> to the final CLR type.");

Match PrimitiveDataFrameColumn<T> to the final CLR type.