02 - Explore, Select & Filter - C#

Explore, Select & Filter - C#

Quote

“If we have data, let’s look at data. If all we have are opinions, let’s go with mine.”

— Jim Barksdale

Summary

Covers the full dataframe inspection and subsetting workflow in C#, using Polars.NET and Microsoft.Data.Analysis to preview data, profile columns, narrow schemas, and isolate rows on real EuroStoxx inputs. The note exists to establish the basic read-first discipline for tabular work: inspect shape, types, nulls, and distributions before writing transforms, and understand where Polars expressions stay composable while MDA falls back to explicit boolean-column mechanics.

Data Exploration

• Preview datasets with Head, Tail, and sample-style views, then inspect shape, schema, and general structure before narrowing the frame
• Run descriptive statistics, null audits, value counts, distinct-count checks, and reusable profiling helpers to understand numeric ranges, categorical distributions, and missing-data hotspots
• Select columns by name, index, and pattern, keeping only the fields needed for downstream work and understanding the difference between whole-frame selection and single-column references
• Filter rows with comparison expressions, combined predicates, and membership tests, contrasting Polars.NET expression trees with MDA’s explicit PrimitiveDataFrameColumn<bool> mask pattern

Operations and safety

• Warnings: MDA Filter requires a boolean column instead of an inline expression, and MDA string-index selection returns a live column reference rather than a copy
• Recommendations: 4 practices covering profiling before transforms, filtering early, preferring Polars.NET expressions for complex predicates, and checking column existence before selection
• Troubleshooting: 3 failure modes covering missing columns, empty filter results from restrictive or mistyped predicates, and MDA Description() omitting non-numeric columns

Glossary

Head / Tail

• Methods that return the first or last n rows of a dataframe for quick inspection.

• They matter because the note starts with these previews to confirm load quality, row ordering, and schema shape before any deeper analysis.

Preview, not view

These calls return new dataframe objects for display and inspection. They are not interactive windows into the original frame.

---

Sample

• A random subset of rows used to inspect representative records without reading the full frame from top or bottom only.

• It matters because exploratory work often needs a spot-check that is not biased toward early or late rows.

MDA sampling is manual

In this note, MDA sampling is built through random row-index selection plus a boolean mask, which is the same pattern you later reuse for more complex row filters.

---

Describe / Description()

• Summary-statistics helpers that compute high-level metrics such as min, max, mean, and other descriptive values over columns.

• They matter because these methods are the fastest first-pass signal for impossible ranges, suspicious sparsity, and distribution shape.

Output shapes differ

MDA Description() returns a dataframe whose statistics are often string-shaped for display, so downstream numeric reuse may require parsing rather than direct arithmetic.

---

Info()

• A schema-inspection helper that reports column metadata such as names, types, and basic completeness characteristics.

• It matters because the note uses it as the structural counterpart to statistical profiling before selection or filtering begins.

Structure before logic

Info() will not tell you whether values are analytically correct, but it quickly exposes whether the frame loaded with the columns and CLR types you expected.

---

Select

• The operation that narrows a dataframe to a chosen subset of columns.

• It matters because schema reduction is one of the cheapest ways to make downstream logic clearer, safer, and lighter on memory.

API symmetry is limited

Polars.NET Select returns a new frame, while MDA often uses indexers or column collections directly. Similar intent does not imply identical object behavior.

---

Filter

• The row-subsetting operation that keeps only records matching a condition.

• It matters because nearly every analytical workflow depends on turning a large frame into the exact subset relevant to the next calculation.

Predicate input differs by library

Polars.NET accepts composable expressions such as Col("x").Gt(5). MDA expects an explicit boolean column whose length matches the dataframe.

---

Boolean mask

• A boolean vector with one element per row, where true means keep the row and false means discard it.

• It matters because MDA filtering, manual sampling, and many reusable selection patterns in the note depend on building masks explicitly.

Length and alignment matter

A mask is only valid when it matches the dataframe row count exactly. If the lengths drift, the filter is invalid no matter how correct the logic seems.

---

Membership test

• A filter condition that checks whether a value belongs to a specified set of allowed candidates.

• It matters because real filtering work often uses symbol lists, sector sets, or categorical whitelists rather than single-value comparisons.

Set-based filtering scales better mentally

Once conditions stop being simple equality checks, membership tests are usually easier to read and maintain than long chains of or comparisons.

---

NullCount

• A per-column measure of how many missing values are present.

• It matters because nulls change comparison behavior, descriptive statistics, and downstream transformation assumptions long before a pipeline throws an exception.

Missingness is column-local information

NullCount tells you which fields are sparse, but not whether those nulls are acceptable business logic or a failed ingestion step.

---

ValueCounts()

• A column-level operation that returns each distinct value together with its frequency.

• It matters because categorical distributions are one of the fastest ways to spot unexpected codes, rare outliers, or mislabeled data.

Frequency is often more useful than uniqueness

A distinct list shows what exists. ValueCounts() also shows what dominates, what is rare, and what may have gone wrong operationally.

---

Distinct / unique count

• The set of unique values in a column, or the count of how many such values exist.

• It matters because uniqueness checks tell you whether identifier columns, dimensions, or categorical domains behave as expected.

MDA often reaches for LINQ here

In MDA, distinct-value workflows commonly involve casting and LINQ Distinct() rather than a single dataframe-native method that mirrors Polars exactly.

---

Column reference

• A handle to an existing column object rather than an isolated copy of its values.

• It matters because MDA string-based column access returns a live reference, which affects whether later mutation changes the original dataframe.

References propagate mutation

If you treat an MDA column reference like a detached slice, you can accidentally modify shared state and misread the result of subsequent exploration steps.

---

Suppress CS1701/CS1702 assembly version warnings in .NET Interactive. Run this cell once before any cells that use NuGet packages.

using System.Reflection;
using Microsoft.DotNet.Interactive;
using Microsoft.DotNet.Interactive.CSharp;
 
var csharpKernel = (CSharpKernel)Kernel.Root.FindKernelByName("csharp");
var optionsField = typeof(CSharpKernel).GetField("_scriptOptions",
    BindingFlags.NonPublic | BindingFlags.Instance);
 
var scriptOptions = optionsField.GetValue(csharpKernel);
var withWarningLevel = scriptOptions.GetType().GetMethod("WithWarningLevel");
var newOptions = withWarningLevel.Invoke(scriptOptions, new object[] { 0 });
optionsField.SetValue(csharpKernel, newOptions);

Install NuGet packages and import namespaces. Alias Microsoft.Data.Analysis as MDA so DataFrame continues to refer to Polars.NET inside mixed examples.

#r "nuget: Polars.NET, 0.4.0"
#r "nuget: Polars.NET.Native.win-x64, 0.4.0"
#r "nuget: Microsoft.Data.Analysis, 0.23.0"
 
using System.IO;
using System.Linq;
using System.Collections.Generic;
using System.Globalization;
using System.Text.RegularExpressions;
using Polars.CSharp;
using static Polars.CSharp.Polars;
using MDA = Microsoft.Data.Analysis;
using Microsoft.DotNet.Interactive.Formatting;
 
Formatter.Register<Polars.CSharp.DataFrame>((df, writer) =>
{
    var html = df.ToHtml();
    html = System.Text.RegularExpressions.Regex.Replace(html, @"(&gt;|>)(.+?)(&lt;|<)", @"$1$2$3");
    html = System.Text.RegularExpressions.Regex.Replace(html, @">""(.+?)""<", @">$1<");
    writer.Write(html);
}, "text/html");
Formatter.Register<Polars.CSharp.Series>((s, writer) =>
    writer.Write($"<pre style='font-size:14px'>{s}</pre>"), "text/html");
 
var DATA = Path.Combine("..", "data");
Console.WriteLine($"Data directory: {Path.GetFullPath(DATA)}");
 
static Type[] OhlcvCsvTypes() => new[]
{
    typeof(long), typeof(string), typeof(DateTime), typeof(decimal), typeof(decimal),
    typeof(decimal), typeof(decimal), typeof(decimal), typeof(long), typeof(decimal), typeof(decimal), typeof(bool)
};
 
static decimal? ToNullableDecimal(object value)
{
    if (value is null) return null;
    if (value is decimal d) return d;
    if (value is string s)
    {
        if (string.IsNullOrWhiteSpace(s)) return null;
        return decimal.Parse(s, NumberStyles.Float, CultureInfo.InvariantCulture);
    }
    return Convert.ToDecimal(value, CultureInfo.InvariantCulture);
}
 
static MDA.DataFrame ConvertColumnsToDecimal(MDA.DataFrame df, params string[] columnNames)
{
    foreach (var columnName in columnNames)
    {
        var index = df.Columns.IndexOf(columnName);
        if (index < 0) continue;
        var source = df.Columns[index];
        var target = new MDA.PrimitiveDataFrameColumn<decimal>(columnName, df.Rows.Count);
        for (long row = 0; row < df.Rows.Count; row++)
        {
            var value = ToNullableDecimal(source[row]);
            if (value.HasValue) target[row] = value.Value;
        }
        df.Columns.Remove(columnName);
        df.Columns.Insert(index, target);
    }
    return df;
}
 
static MDA.DataFrame LoadOhlcvCsv(string dataDir) =>
    MDA.DataFrame.LoadCsv(Path.Combine(dataDir, "eurostoxx50_ohlcv.csv"), dataTypes: OhlcvCsvTypes());
 
static MDA.DataFrame LoadScoresDailyCsv(string dataDir)
{
    var df = MDA.DataFrame.LoadCsv(Path.Combine(dataDir, "scores_daily.csv"));
    return ConvertColumnsToDecimal(
        df,
        "pe_zscore", "pb_zscore", "ev_ebitda_zscore", "yield_zscore", "relative_value_score",
        "relative_strength", "sma_50_ratio", "sma_200_ratio", "dist_from_52w_high", "momentum_score",
        "implied_upside", "recommendation_mean", "sentiment_score", "composite_score", "sma_30_close",
        "sma_90_close", "market_cap", "index_weight", "current_price", "day_change_pct",
        "five_day_change_pct", "ytd_change_pct"
    );
}

Data directory: c:\Users\aperi\DEV\LANG\data

Load the primary datasets used throughout this notebook. Both libraries read the same CSV files, so the rest of the page compares API shape and workflow rather than data differences.

var dfP = Polars.CSharp.DataFrame.ReadCsv(Path.Combine(DATA, "eurostoxx50_ohlcv.csv"), tryParseDates: true);
var dimP = Polars.CSharp.DataFrame.ReadCsv(Path.Combine(DATA, "dim_country.csv"));
 
display($"OHLCV: {dfP.Shape}  |  DimCountry: {dimP.Shape}");
 
var df = LoadOhlcvCsv(DATA);
var dim = MDA.DataFrame.LoadCsv(Path.Combine(DATA, "dim_country.csv"));
 
display($"OHLCV: ({df.Rows.Count}, {df.Columns.Count})  |  DimCountry: ({dim.Rows.Count}, {dim.Columns.Count})");

OHLCV: (66355, 12)  |  DimCountry: (212, 2)
OHLCV: (66355, 12)  |  DimCountry: (212, 2)

Current API and execution-model check | 2026-04

Microsoft’s DataFrame API documents Head, Tail, Description, Info, Filter, OrderBy, LoadCsv, and IDataView interoperability. Polars’ lazy optimization guide documents predicate, projection, and slice pushdown. In practice, the same explore/select/filter logic can stay closer to an optimizable query plan in Polars, while MDA remains an eager in-memory dataframe API.

---

Data Exploration

The first step after loading data is exploration: previewing rows, inspecting schema, profiling nulls, and understanding value distributions before you write downstream transformations. Polars.NET tends to surface these operations as concise dataframe methods; Microsoft.Data.Analysis exposes the same work through managed DataFrame and DataFrameColumn APIs.

Head, Tail, and Sample

Polars.NET | Preview first and last rows with Head and Tail

.Head(n) and .Tail(n) return the first and last N rows. .Sample(n) returns N random rows. These are the most common entry points for data exploration.

Calls Head(5) and Tail(5) on the 66,355-row OHLCV DataFrame, displaying 5-row previews from each end to confirm schema, date range (2021-01-04 to 2026-03-12), and that all 12 columns are present.

display("Head(5):");
display(dfP.Head(5));
display("Tail(5):");
dfP.Tail(5)

Head(5):

id	symbol	date	open	high	low	close	adj_close	volume	dividends	stock_splits	is_filled
21160	ABI.BR	2021-01-04	58.15	58.85	56.78	57.21	53.5761	1513937	0	0	false
21161	ABI.BR	2021-01-05	56.9	57.98	56.75	57.18	53.548	1382722	0	0	false
21162	ABI.BR	2021-01-06	57.96	58.94	57.39	58.77	55.037	1370204	0	0	false
21163	ABI.BR	2021-01-07	58.68	58.86	57.88	58.4	54.6905	1469911	0	0	false
21164	ABI.BR	2021-01-08	58.16	58.4	57.43	57.86	54.1848	1428681	0	0	false

Tail(5):

id	symbol	date	open	high	low	close	adj_close	volume	dividends	stock_splits	is_filled
64828	WKL.AS	2026-03-06	69.02	69.36	67.82	68.52	68.52	1143729	0	0	false
66875	WKL.AS	2026-03-09	68.78	69.16	67.64	68.64	68.64	841503	0	0	false
66876	WKL.AS	2026-03-10	68.8	69.16	66.34	67.16	67.16	1355645	0	0	false
66877	WKL.AS	2026-03-11	67.5	69.6	67.02	67.22	67.22	1142531	0	0	false
66929	WKL.AS	2026-03-12	67	67.54	66.28	67.32	67.32	210379	0	0	false

Microsoft.Data.Analysis | Preview first and last rows with Head and Tail

Microsoft.Data.Analysis exposes Head(n) and Tail(n) directly on DataFrame, so the first preview workflow is very close to Polars for this use case.

Calls Head(5) and Tail(5) on the 66,355-row OHLCV DataFrame to verify that the dataset loaded correctly, the row ordering is intact, and all 12 columns are present in both previews.

// Microsoft.Data.Analysis — Head / Tail
display("Head(5):");
display(df.Head(5));
display("Tail(5):");
df.Tail(5)

Head(5):

id	symbol	date	open	high	low	close	adj_close	volume	dividends	stock_splits	is_filled
21160	ABI.BR	2021-01-04 00:00:00Z	58.15	58.85	56.78	57.21	53.5761	1513937	0.0	0.0	False
21161	ABI.BR	2021-01-05 00:00:00Z	56.9	57.98	56.75	57.18	53.548	1382722	0.0	0.0	False
21162	ABI.BR	2021-01-06 00:00:00Z	57.96	58.94	57.39	58.77	55.037	1370204	0.0	0.0	False
21163	ABI.BR	2021-01-07 00:00:00Z	58.68	58.86	57.88	58.4	54.6905	1469911	0.0	0.0	False
21164	ABI.BR	2021-01-08 00:00:00Z	58.16	58.4	57.43	57.86	54.1848	1428681	0.0	0.0	False

Tail(5):

id	symbol	date	open	high	low	close	adj_close	volume	dividends	stock_splits	is_filled
64828	WKL.AS	2026-03-06 00:00:00Z	69.02	69.36	67.82	68.52	68.52	1143729	0.0	0.0	False
66875	WKL.AS	2026-03-09 00:00:00Z	68.78	69.16	67.64	68.64	68.64	841503	0.0	0.0	False
66876	WKL.AS	2026-03-10 00:00:00Z	68.8	69.16	66.34	67.16	67.16	1355645	0.0	0.0	False
66877	WKL.AS	2026-03-11 00:00:00Z	67.5	69.6	67.02	67.22	67.22	1142531	0.0	0.0	False
66929	WKL.AS	2026-03-12 00:00:00Z	67.0	67.54	66.28	67.32	67.32	210379	0.0	0.0	False

Microsoft.Data.Analysis | Random sample with an explicit mask

The current API surface documents Sample(Int32), but the explicit helper below is still useful because it shows the row-filtering model MDA falls back to in more complex cases: build a boolean mask and pass it into Filter().

Builds a Sample(DataFrame data, int n) helper that randomly selects row indices, turns them into a boolean mask, and returns data.Filter(mask) to produce a 5-row exploratory sample.

// Microsoft.Data.Analysis — Sample(n) returns n random rows
// (No native method, using boolean mask generation)
DataFrame Sample(DataFrame data, int n)
{
    var rand = new Random();
    var indices = new HashSet<long>();
    while(indices.Count < n && indices.Count < data.Rows.Count)
    {
        indices.Add(rand.NextInt64(0, data.Rows.Count));
    }
 
    var mask = new PrimitiveDataFrameColumn<bool>("mask", data.Rows.Count);
    for(long i = 0; i < data.Rows.Count; i++) mask[i] = indices.Contains(i);
 
    return data.Filter(mask);
}
 
Sample(df, 5)

id	symbol	date	open	high	low	close	adj_close	volume	dividends	stock_splits	is_filled
62110	ADS.DE	2021-02-03 00:00:00Z	279.9	279.9	274.2	275.0	262.7958	405362	0.0	0.0	False
47784	BMW.DE	2021-11-24 00:00:00Z	94.23	94.83	91.81	92.49	70.7878	1403839	0.0	0.0	False
65828	DSY.PA	2024-11-21 00:00:00Z	32.17	32.28	31.83	32.17	31.921	845398	0.0	0.0	False
27022	EL.PA	2023-03-24 00:00:00Z	158.5725	158.915	156.8601	158.6215	150.249	458988	0.0	0.0	False
43193	ENI.MI	2024-06-21 00:00:00Z	14.06	14.156	13.906	13.936	12.5348	29704326	0.0	0.0	False

Microsoft.Data.Analysis | Inspect shape with row counts and Info

MDA exposes row and column counts directly from Rows.Count and Columns.Count, and Info() provides the concise schema-like view used for quick inspection.

Prints the OHLCV shape as (66355, 12) and then calls Info() to summarize the 12 columns, their CLR data types, and basic completeness metadata before any downstream selection or filtering logic.

// Microsoft.Data.Analysis — Shape and schema
display($"Shape: ({df.Rows.Count}, {df.Columns.Count})  |  Height: {df.Rows.Count}  |  Width: {df.Columns.Count}");
df.Info();

Shape: (66355, 12)  |  Height: 66355  |  Width: 12

Microsoft.Data.Analysis | Summary statistics with Description

Description() is the MDA equivalent for numeric summary statistics. It returns another DataFrame, which keeps the result easy to inspect or reuse in notebook workflows.

Computes descriptive statistics for the numeric columns in df, returning a summary DataFrame that highlights counts, minima, maxima, and mean values for the OHLCV data.

// Microsoft.Data.Analysis — Description() returns a summary DataFrame
df.Description()

Description	id	date	open	high	low	close	adj_close	volume	dividends	stock_splits
Length (excluding null values)	66355	66355	66355	66355	66355	66355	66355	66355	66355	66355
Max	66930	<null>	2926	2957	2813	2839	2802.9382	376391550	22.5	5
Min	1	<null>	1.601	1.6628	1.5842	1.6066	1.2013	0	0	0
Mean	33179.734	<null>	197.04053	199.36412	194.58578	197.0349	190.49492	5942123.5	0.011756673	0.00017203268

Microsoft.Data.Analysis | Count nulls per column

Null counts live on each DataFrameColumn via NullCount, so the standard pattern is to iterate columns and record the fields that actually contain missing data.

Loads scores_daily.csv, iterates through its columns, and prints only the fields whose NullCount is greater than zero so that missing-value hot spots stand out immediately.

// Microsoft.Data.Analysis — NullCount is a property on DataFrameColumn
// Use scores_daily which has real nulls
var scP = LoadScoresDailyCsv(DATA);
Console.WriteLine($"scores_daily: ({scP.Rows.Count}, {scP.Columns.Count})");
foreach (var col in scP.Columns)
{
    var nc = col.NullCount;
    if (nc > 0)
        Console.WriteLine($"  {col.Name,-28} {nc,4} nulls");
}

scores_daily: (466, 36)
  pe_zscore                       3 nulls
  pb_zscore                       6 nulls
  ev_ebitda_zscore               71 nulls
  yield_zscore                   35 nulls
  recommendation_mean            14 nulls

Microsoft.Data.Analysis | Frequency distribution with ValueCounts

ValueCounts() is available on a column and returns a new DataFrame of unique values plus their counts. This is the fastest way to inspect categorical distributions in MDA.

Computes ValueCounts() for the symbol column, returning one row per ticker together with its observation count across the 66,355-row OHLCV dataset.

// Microsoft.Data.Analysis — ValueCounts() on a Column returns a DataFrame
df.Columns["symbol"].ValueCounts()

Values	Counts
ABI.BR	1331
AD.AS	1331
ADS.DE	1324
ADYEN.AS	1331
AI.PA	1331
AIR.PA	1331
ALV.DE	1324
ARGX.BR	1331
ASML.AS	1331
BAS.DE	1324
BAYN.DE	1324
BBVA.MC	1329
BMW.DE	1324
BN.PA	1331
BNP.PA	1331
CS.PA	1331
DB1.DE	1324
DG.PA	1331
DHL.DE	1324
DSY.PA	1331
DTE.DE	1324
EL.PA	1331
ENEL.MI	1321
ENI.MI	1321
ENR.DE	1324
IBE.MC	1329
IFX.DE	1324
INGA.AS	1331
ISP.MI	1321
ITX.MC	1329
MBG.DE	1324
MC.PA	1331
MUV2.DE	1324
NDA-FI.HE	1306
OR.PA	1331
PRX.AS	1331
RACE.MI	1321
RHM.DE	1324
RMS.PA	1331
SAF.PA	1331
SAN.MC	1329
SAN.PA	1331
SAP.DE	1324
SGO.PA	1331
SIE.DE	1324
SU.PA	1331
TTE.PA	1331
UCG.MI	1321
VOW.DE	1324
WKL.AS	1331

Microsoft.Data.Analysis | Distinct values with LINQ Distinct

When you want the set of unique values without the counts, the MDA pattern is usually to cast the column to a typed enumerable and use LINQ Distinct() before materializing the result back into a DataFrame.

Casts the symbol column to string, computes the distinct tickers with LINQ, prints the unique-count result, and materializes the distinct values into a one-column DataFrame.

// Microsoft.Data.Analysis — Unique counting and distinct values using LINQ
var symSeries = df.Columns["symbol"].Cast<string>();
var distinctSymbols = symSeries.Distinct().ToArray();
display($"NUnique: {distinctSymbols.Length}");
 
new DataFrame(new StringDataFrameColumn("symbol", distinctSymbols))

NUnique: 50

symbol
ABI.BR
AD.AS
ADS.DE
ADYEN.AS
AI.PA
AIR.PA
ALV.DE
ARGX.BR
ASML.AS
BAS.DE
BAYN.DE
BBVA.MC
BMW.DE
BN.PA
BNP.PA
CS.PA
DB1.DE
DG.PA
DHL.DE
DSY.PA
DTE.DE
EL.PA
ENEL.MI
ENI.MI
ENR.DE
IBE.MC
IFX.DE
INGA.AS
ISP.MI
ITX.MC
MBG.DE
MC.PA
MUV2.DE
NDA-FI.HE
OR.PA
PRX.AS
RACE.MI
RHM.DE
RMS.PA
SAF.PA
SAN.MC
SAN.PA
SAP.DE
SGO.PA
SIE.DE
SU.PA
TTE.PA
UCG.MI
VOW.DE
WKL.AS

Microsoft.Data.Analysis | Memory estimate via CLR-type heuristic

MDA does not expose a direct equivalent to Polars EstimatedSize(), so a practical approximation is to estimate bytes from the inferred CLR type and multiply by column length.

Estimates dataframe size by summing column length x assumed byte width across all columns, then reports the approximate byte and megabyte footprint alongside the dataframe shape.

// Microsoft.Data.Analysis — estimate size
long estBytes = 0;
foreach (var col in df.Columns)
{
    estBytes += col.Length * (col.DataType.Name switch
    {
        "Int32" or "Single" => 4,
        "Int64" or "Double" or "DateTime" => 8,
        "Decimal" => 16,
        "Boolean" => 1,
        "String" => 40,
        _ => 8
    });
}
display($"Estimated size: {estBytes:N0} bytes ({estBytes / 1_048_576.0:F2} MB)");
display($"Shape: ({df.Rows.Count}, {df.Columns.Count})  |  {df.Rows.Count:N0} rows x {df.Columns.Count} cols");

Estimated size: 11'744'835 bytes (11.20 MB)

Shape: (66355, 12)  |  66'355 rows x 12 cols

Microsoft.Data.Analysis | Reusable quick-profile function

When basic exploration needs to become repeatable, build a small profiling function that emits column name, CLR type, null count, and distinct count for any DataFrame.

Defines ProfileDataFrame(DataFrame d), computes per-column type, null count, and unique count, and applies it to both eurostoxx50_ohlcv and dim_country to produce reusable profile tables.

// Microsoft.Data.Analysis — reusable profiler that returns a summary DataFrame
DataFrame ProfileDataFrame(DataFrame d)
{
    var cols = d.Columns.ToArray();
    var colNames = new string[cols.Length];
    var types = new string[cols.Length];
    var nulls = new long[cols.Length];
    var uniques = new long[cols.Length];
 
    for (int j = 0; j < cols.Length; j++)
    {
        var s = d.Columns[j];
        colNames[j] = s.Name;
        types[j] = s.DataType.Name;
        nulls[j] = s.NullCount;
        uniques[j] = s.ValueCounts().Rows.Count;
    }
 
    return new DataFrame(
        new StringDataFrameColumn("column", colNames),
        new StringDataFrameColumn("type", types),
        new PrimitiveDataFrameColumn<long>("nulls", nulls),
        new PrimitiveDataFrameColumn<long>("unique", uniques)
    );
}
 
display($"eurostoxx50_ohlcv: ({df.Rows.Count}, {df.Columns.Count})");
display(ProfileDataFrame(df));
display($"dim_country: ({dim.Rows.Count}, {dim.Columns.Count})");
display(ProfileDataFrame(dim));

eurostoxx50_ohlcv: (66355, 12)

column	type	nulls	unique
id	Int64	0	66355
symbol	String	0	50
date	DateTime	0	1331
open	Decimal	0	29671
high	Decimal	0	31651
low	Decimal	0	31695
close	Decimal	0	31505
adj_close	Decimal	0	57739
volume	Int64	0	65199
dividends	Decimal	0	216
stock_splits	Decimal	0	6
is_filled	Boolean	0	2

dim_country: (212, 2)

column	type	nulls	unique
country_name	String	0	212
iso_alpha2	String	0	212

Microsoft.Data.Analysis | Select a single column by name

Selecting one column returns a DataFrameColumn, not a one-column DataFrame. That is convenient for direct vector operations, but downstream code needs to stay clear about whether it expects a column or a frame.

Selects close and symbol from df.Columns[...], then prints each column’s name, length, and data type metadata to confirm the object returned is a DataFrameColumn.

// Microsoft.Data.Analysis — single column returns a DataFrameColumn
var closeSeries = df.Columns["close"];
display($"Name: {closeSeries.Name}  |  Length: {closeSeries.Length}  |  Type: {closeSeries.DataType.Name}");
 
var symbolSeries = df.Columns["symbol"];
display($"Name: {symbolSeries.Name}  |  Length: {symbolSeries.Length}");

Name: close  |  Length: 66355  |  Type: Decimal

Name: symbol  |  Length: 66355

Microsoft.Data.Analysis | Select multiple columns into a new DataFrame

The standard MDA pattern is to construct a new DataFrame from the selected column objects. This is explicit and readable, but less declarative than Polars projection expressions when the selection logic becomes dynamic.

Builds a new DataFrame from date, symbol, close, and volume, then previews the first five rows to confirm the four-column projection.

// Microsoft.Data.Analysis — Select specific columns to form a new DataFrame
var selected = new DataFrame(df.Columns["date"], df.Columns["symbol"], df.Columns["close"], df.Columns["volume"]);
selected.Head(5)

date	symbol	close	volume
2021-01-04 00:00:00Z	ABI.BR	57.21	1513937
2021-01-05 00:00:00Z	ABI.BR	57.18	1382722
2021-01-06 00:00:00Z	ABI.BR	58.77	1370204
2021-01-07 00:00:00Z	ABI.BR	58.4	1469911
2021-01-08 00:00:00Z	ABI.BR	57.86	1428681

Microsoft.Data.Analysis | Compute columns explicitly and rename with SetName

MDA supports vectorized column arithmetic, but computed projections are assembled explicitly: clone the source columns you want to keep, compute the derived column, rename it, and build a new DataFrame from those pieces.

Clones symbol and close, renames close to price, computes range = high - low, and materializes the result as a three-column DataFrame for inspection.

// Microsoft.Data.Analysis — compute-on-select and renaming
var symbolCol = df.Columns["symbol"].Clone();
var priceCol = df.Columns["close"].Clone();
priceCol.SetName("price");
var highCol = (PrimitiveDataFrameColumn<decimal>)df.Columns["high"];
var lowCol = (PrimitiveDataFrameColumn<decimal>)df.Columns["low"];
var rangeCol = new PrimitiveDataFrameColumn<decimal>("range", df.Rows.Count);
for (long i = 0; i < df.Rows.Count; i++) if (highCol[i].HasValue && lowCol[i].HasValue) rangeCol[i] = highCol[i].Value - lowCol[i].Value;
var computed = new DataFrame(symbolCol, priceCol, rangeCol);
computed.Head(5)

symbol	price	range
ABI.BR	57.21	2.07
ABI.BR	57.18	1.23
ABI.BR	58.77	1.55
ABI.BR	58.4	0.98
ABI.BR	57.86	0.97

Microsoft.Data.Analysis | Exclude columns by building the keep-set

Dropping columns is usually written as the inverse problem: define the columns to exclude, then build a new DataFrame from the columns that remain.

Defines a dropCols set, selects every column not in that set, prints the resulting column list, and previews the trimmed frame.

// Microsoft.Data.Analysis — Drop columns by selecting the ones to keep
var dropCols = new HashSet<string> { "id", "adj_close", "dividends", "stock_splits", "is_filled" };
var keepCols = df.Columns.Where(c => !dropCols.Contains(c.Name)).ToArray();
var trimmed = new DataFrame(keepCols);
display($"Columns: {string.Join(", ", trimmed.Columns.Select(c => c.Name))}");
trimmed.Head(3)

Columns: symbol, date, open, high, low, close, volume

symbol	date	open	high	low	close	volume
ABI.BR	2021-01-04 00:00:00Z	58.15	58.85	56.78	57.21	1513937
ABI.BR	2021-01-05 00:00:00Z	56.9	57.98	56.75	57.18	1382722
ABI.BR	2021-01-06 00:00:00Z	57.96	58.94	57.39	58.77	1370204

Microsoft.Data.Analysis | Select columns matching a regex pattern

Regex-based selection in MDA is a Columns.Where(...) operation followed by a new DataFrame constructor. This keeps the logic flexible when column names follow domain conventions.

Uses the regex ^(open|high|low|close)$ to isolate the OHLC price columns, prints the matched column names, and previews the resulting four-column DataFrame.

// Microsoft.Data.Analysis — select columns by regex (match OHLC price columns)
var pattern = new Regex("^(open|high|low|close)$");
var priceCols = df.Columns.Where(c => pattern.IsMatch(c.Name)).ToArray();
var regexDf = new DataFrame(priceCols);
display($"Matched: {string.Join(", ", regexDf.Columns.Select(c => c.Name))}");
regexDf.Head(5)

Matched: open, high, low, close

open	high	low	close
58.15	58.85	56.78	57.21
56.9	57.98	56.75	57.18
57.96	58.94	57.39	58.77
58.68	58.86	57.88	58.4
58.16	58.4	57.43	57.86

Microsoft.Data.Analysis | Rename columns with cloned columns and SetName

Renaming usually means cloning the source columns, mutating their names, and then materializing the renamed set into a new frame.

Clones symbol, close, and volume, renames them to ticker, price, and vol, prints the new schema, and previews the renamed DataFrame.

// Microsoft.Data.Analysis — Rename via SetName on cloned columns
var renamed = new DataFrame(df.Columns["symbol"].Clone(), df.Columns["close"].Clone(), df.Columns["volume"].Clone());
renamed.Columns["symbol"].SetName("ticker");
renamed.Columns["close"].SetName("price");
renamed.Columns["volume"].SetName("vol");
 
display($"Columns: {string.Join(", ", renamed.Columns.Select(c => c.Name))}");
renamed.Head(3)

Columns: ticker, price, vol

ticker	price	vol
ABI.BR	57.21	1513937
ABI.BR	57.18	1382722
ABI.BR	58.77	1370204

Microsoft.Data.Analysis | Reorder columns by constructor order

Column order is simply the order in which the selected columns are passed into the new DataFrame. That makes reordering straightforward, but still a manual projection step.

Constructs a new DataFrame in the order symbol, date, volume, open, high, low, close, prints the resulting column order, and previews the reordered frame.

// Microsoft.Data.Analysis — Reorder by selecting them in order into a new DataFrame
var reordered = new DataFrame(
    df.Columns["symbol"], df.Columns["date"], df.Columns["volume"],
    df.Columns["open"], df.Columns["high"], df.Columns["low"], df.Columns["close"]
);
display($"Column order: {string.Join(", ", reordered.Columns.Select(c => c.Name))}");
reordered.Head(3)

Column order: symbol, date, volume, open, high, low, close

symbol	date	volume	open	high	low	close
ABI.BR	2021-01-04 00:00:00Z	1513937	58.15	58.85	56.78	57.21
ABI.BR	2021-01-05 00:00:00Z	1382722	56.9	57.98	56.75	57.18
ABI.BR	2021-01-06 00:00:00Z	1370204	57.96	58.94	57.39	58.77

Microsoft.Data.Analysis | Boolean filter with elementwise comparison

Numeric predicates are expressed as vectorized elementwise comparisons on a typed column, and the resulting boolean column is passed into DataFrame.Filter(...).

Casts close to MDA.PrimitiveDataFrameColumn<decimal>, builds the predicate close > 500, filters the dataframe, prints the row count, and previews the first five matching rows.

// Microsoft.Data.Analysis — Filter with elementwise expressions
var closeCol = (PrimitiveDataFrameColumn<decimal>)df.Columns["close"];
var expensiveMask = new PrimitiveDataFrameColumn<bool>("close_gt_500", df.Rows.Count);
for (long i = 0; i < df.Rows.Count; i++) if (closeCol[i].HasValue) expensiveMask[i] = closeCol[i].Value > 500.0m;
var expensive = df.Filter(expensiveMask);
display($"Rows where close > 500: {expensive.Rows.Count}");
expensive.Head(5)

Rows where close > 500: 6155

id	symbol	date	open	high	low	close	adj_close	volume	dividends	stock_splits	is_filled
60763	ADYEN.AS	2021-01-04 00:00:00Z	1900.0	1921.5	1856.0	1859.5	1859.5	99408	0.0	0.0	False
60764	ADYEN.AS	2021-01-05 00:00:00Z	1848.5	1857.0	1814.0	1829.0	1829.0	86256	0.0	0.0	False
60765	ADYEN.AS	2021-01-06 00:00:00Z	1822.0	1824.0	1706.5	1733.0	1733.0	156844	0.0	0.0	False
60766	ADYEN.AS	2021-01-07 00:00:00Z	1735.0	1754.0	1708.5	1714.5	1714.5	90183	0.0	0.0	False
60767	ADYEN.AS	2021-01-08 00:00:00Z	1730.0	1764.5	1715.0	1756.5	1756.5	97176	0.0	0.0	False

Microsoft.Data.Analysis | Compound filters with And, Or, and explicit negation

Compound filters are composed from boolean columns with .And() and .Or(). In practice, the mask construction is explicit, which keeps the logic readable but gets verbose faster than a Polars predicate chain.

Builds boolean masks for ASML.AS, SAP.DE, and close > 600, then evaluates ASML AND close > 600, ASML OR SAP, and a non-ASML filter to show the three common boolean compositions.

// Microsoft.Data.Analysis — AND / OR / NOT elementwise operators
var symbolColStr = df.Columns["symbol"];
var isAsml = symbolColStr.ElementwiseEquals("ASML.AS");
var isSap = symbolColStr.ElementwiseEquals("SAP.DE");
var closeOver600 = new PrimitiveDataFrameColumn<bool>("close_over_600", df.Rows.Count);
for (long i = 0; i < df.Rows.Count; i++) if (closeCol[i].HasValue) closeOver600[i] = closeCol[i].Value > 600.0m;
var filtered = df.Filter((PrimitiveDataFrameColumn<bool>)isAsml.And(closeOver600));
display($"ASML AND close > 600: {filtered.Rows.Count} rows");
display(filtered.Head(3));
var orFilter = df.Filter((PrimitiveDataFrameColumn<bool>)isAsml.Or(isSap));
display($"ASML OR SAP: {orFilter.Rows.Count} rows");
var notFilter = df.Filter((PrimitiveDataFrameColumn<bool>)isAsml.ElementwiseNotEquals(true));
display($"NOT ASML: {notFilter.Rows.Count} rows");

ASML AND close > 600: 840 rows

id	symbol	date	open	high	low	close	adj_close	volume	dividends	stock_splits	is_filled
136	ASML.AS	2021-07-14 00:00:00Z	599.5	611.8	597.2	609.1	582.9708	641585	0.0	0.0	False
142	ASML.AS	2021-07-22 00:00:00Z	610.0	625.9	608.2	620.8	594.169	788099	0.0	0.0	False
143	ASML.AS	2021-07-23 00:00:00Z	622.9	639.0	617.5	638.8	611.3967	833737	0.0	0.0	False

ASML OR SAP: 2655 rows

NOT ASML: 65024 rows

Microsoft.Data.Analysis | Filter by membership with a manual boolean mask

Membership filtering is usually implemented by testing each value against a HashSet<T> and writing the result into a boolean mask column.

Builds a ticker set for ASML.AS, SAP.DE, and SIE.DE, evaluates membership row by row into a boolean mask, filters the dataframe, and previews the first five matching rows.

// Microsoft.Data.Analysis — IsIn requires manual evaluation into a boolean mask
var techTickers = new HashSet<string> { "ASML.AS", "SAP.DE", "SIE.DE" };
var isInMask = new PrimitiveDataFrameColumn<bool>("mask", df.Rows.Count);
 
// Cast to StringDataFrameColumn instead of using .Cast<string>()
var symbols = (StringDataFrameColumn)df.Columns["symbol"];
 
for(long i = 0; i < df.Rows.Count; i++)
{
    // Now indexing with [i] works correctly
    isInMask[i] = techTickers.Contains(symbols[i]);
}
 
var techRows = df.Filter(isInMask);
display($"Tech tickers: {techRows.Rows.Count} rows");
techRows.Head(5)

Tech tickers: 3979 rows

id	symbol	date	open	high	low	close	adj_close	volume	dividends	stock_splits	is_filled
1	ASML.AS	2021-01-04 00:00:00Z	404.0	411.0	402.25	406.25	387.709	789502	0.0	0.0	False
2	ASML.AS	2021-01-05 00:00:00Z	406.55	412.05	401.15	406.9	388.3294	798787	0.0	0.0	False
3	ASML.AS	2021-01-06 00:00:00Z	406.8	407.2	399.2	402.85	384.4644	875711	0.0	0.0	False
4	ASML.AS	2021-01-07 00:00:00Z	404.8	407.8	400.35	403.9	385.4664	874780	0.0	0.0	False
5	ASML.AS	2021-01-08 00:00:00Z	414.25	419.1	413.4	416.05	397.0618	975243	0.0	0.0	False

Microsoft.Data.Analysis | Range filter with composed boolean masks

Range filters are just boolean mask composition: generate the lower-bound and upper-bound comparisons, combine them with .And(), and pass the result into Filter().

Builds a mask for 100 <= close <= 200, filters the dataframe, prints the matching row count, and previews the first five results.

// Microsoft.Data.Analysis — Range filtering using logical And()
var midRangeMask = new PrimitiveDataFrameColumn<bool>("mid_range", df.Rows.Count);
for (long i = 0; i < df.Rows.Count; i++)
{
    if (closeCol[i].HasValue)
    {
        var value = closeCol[i].Value;
        midRangeMask[i] = value >= 100.0m && value <= 200.0m;
    }
}
var midRange = df.Filter(midRangeMask);
display($"Close between 100 and 200: {midRange.Rows.Count} rows");
midRange.Head(5)

Close between 100 and 200: 12013 rows

id	symbol	date	open	high	low	close	adj_close	volume	dividends	stock_splits	is_filled
62387	ADS.DE	2022-03-04 00:00:00Z	196.5	197.64	187.0	187.0	180.5918	1319891	0.0	0.0	False
62388	ADS.DE	2022-03-07 00:00:00Z	177.1	183.4	170.08	176.9	170.8379	2345656	0.0	0.0	False
62389	ADS.DE	2022-03-08 00:00:00Z	172.18	187.06	172.0	184.94	178.6024	1937346	0.0	0.0	False
62391	ADS.DE	2022-03-10 00:00:00Z	211.35	211.8	196.68	197.08	190.3264	1375129	0.0	0.0	False
62415	ADS.DE	2022-04-13 00:00:00Z	198.52	199.74	194.16	197.76	190.9831	755573	0.0	0.0	False

Microsoft.Data.Analysis | Null checks with ElementwiseIsNull and ElementwiseIsNotNull

MDA exposes null predicates as elementwise boolean columns. The main pattern is straightforward: generate the mask, call Filter(), and count or inspect the result.

Filters once for rows with null volume and once for rows with non-null volume, then prints the row counts for each result set.

// Microsoft.Data.Analysis — IsNull / IsNotNull
var withNulls = df.Filter(df.Columns["volume"].ElementwiseIsNull());
display($"Rows with null volume: {withNulls.Rows.Count}");
 
var noNulls = df.Filter(df.Columns["volume"].ElementwiseIsNotNull());
display($"Rows with non-null volume: {noNulls.Rows.Count}");

Rows with null volume: 0

Rows with non-null volume: 66355

Microsoft.Data.Analysis | String predicates through explicit mask evaluation

String filtering often becomes manual mask construction over a MDA.StringDataFrameColumn, especially when you need EndsWith or Contains semantics without a dedicated string-expression namespace.

Builds one mask for Paris-listed symbols ending in .PA and another for symbols containing BN, filters the dataframe with each mask, prints the row counts, and materializes the distinct matching symbols.

// Microsoft.Data.Analysis — string predicates require manual boolean mask evaluation
var isPaMask = new PrimitiveDataFrameColumn<bool>("mask", df.Rows.Count);
var hasBnMask = new PrimitiveDataFrameColumn<bool>("mask", df.Rows.Count);
 
for(long i = 0; i < df.Rows.Count; i++)
{
    var val = symbols[i];
    isPaMask[i] = val != null && val.EndsWith(".PA");
    hasBnMask[i] = val != null && val.Contains("BN");
}
 
var parisStocks = df.Filter(isPaMask);
display($"Paris-listed (.PA): {parisStocks.Rows.Count} rows");
display(new DataFrame(new StringDataFrameColumn("symbol", parisStocks.Columns["symbol"].Cast<string>().Distinct())));
 
var containsB = df.Filter(hasBnMask);
display($"Symbol contains BN: {containsB.Rows.Count} rows");
new DataFrame(new StringDataFrameColumn("symbol", containsB.Columns["symbol"].Cast<string>().Distinct()))

Paris-listed (.PA): 21296 rows

symbol
AI.PA
AIR.PA
BN.PA
BNP.PA
CS.PA
DG.PA
DSY.PA
EL.PA
MC.PA
OR.PA
RMS.PA
SAF.PA
SAN.PA
SGO.PA
SU.PA
TTE.PA

Symbol contains BN: 2662 rows

symbol
BN.PA
BNP.PA

Microsoft.Data.Analysis | Date predicates with an explicit DateTime mask

The date column is evaluated row by row, coercing each value to DateTime where possible and writing the final result into reusable boolean masks. This is clear and robust, but more procedural than Polars’ date-expression API.

Builds one mask for rows in calendar year 2023 and another for Q1 2024, prints the resulting row counts, and previews the first five rows from the Q1 2024 subset.

// Microsoft.Data.Analysis — date predicates evaluated manually
var dates = df.Columns["date"];
var yearMask = new PrimitiveDataFrameColumn<bool>("yearMask", df.Rows.Count);
var q1Mask = new PrimitiveDataFrameColumn<bool>("q1Mask", df.Rows.Count);
 
for(long i = 0; i < df.Rows.Count; i++)
{
    DateTime d;
    if(dates[i] is DateTime dt) d = dt;
    else if (dates[i] is string s && DateTime.TryParse(s, out var p)) d = p;
    else continue;
 
    yearMask[i] = d.Year == 2023;
    q1Mask[i] = d >= new DateTime(2024, 1, 1) && d <= new DateTime(2024, 3, 31);
}
 
var year2023 = df.Filter(yearMask);
display($"Year 2023: {year2023.Rows.Count} rows");
 
var dateRange = df.Filter(q1Mask);
display($"Q1 2024: {dateRange.Rows.Count} rows");
dateRange.Head(5)

Year 2023: 12741 rows

Q1 2024: 3150 rows

id	symbol	date	open	high	low	close	adj_close	volume	dividends	stock_splits	is_filled
21930	ABI.BR	2024-01-02 00:00:00Z	58.72	58.95	58.17	58.77	56.7582	1049145	0.0	0.0	False
21931	ABI.BR	2024-01-03 00:00:00Z	58.64	59.34	58.24	58.37	56.3719	1247000	0.0	0.0	False
21932	ABI.BR	2024-01-04 00:00:00Z	58.36	58.92	58.3	58.81	56.7968	1009526	0.0	0.0	False
21933	ABI.BR	2024-01-05 00:00:00Z	58.26	58.91	58.16	58.86	56.8451	1236000	0.0	0.0	False
21934	ABI.BR	2024-01-08 00:00:00Z	58.47	59.5	58.39	59.38	57.3473	1033238	0.0	0.0	False

Microsoft.Data.Analysis | Access a single row by position

For quick inspection, Head(1) returns a one-row frame while column indexers retrieve specific scalar values from the same row position.

Displays the first row with Head(1), then reads symbol and close from row position 0 via the column indexers to show row-level inspection without materializing a dedicated row object.

// Microsoft.Data.Analysis — single row access
display("Row 0:");
display(df.Head(1));
 
// Access individual values from a row by column indexer
display($"Row 0, symbol: {df.Columns["symbol"][0]}");
display($"Row 0, close:  {df.Columns["close"][0]}");

Row 0:

id	symbol	date	open	high	low	close	adj_close	volume	dividends	stock_splits	is_filled
21160	ABI.BR	2021-01-04 00:00:00Z	58.15	58.85	56.78	57.21	53.5761	1513937	0.0	0.0	False

Row 0, symbol: ABI.BR

Row 0, close:  57.21

Microsoft.Data.Analysis | Slice a row range with a positional mask

The slice pattern shown here creates a positional boolean mask, marks the desired row interval, and filters on that mask.

Builds a mask for row positions 100..104, filters the dataframe on that mask, and displays the resulting five-row slice.

// Microsoft.Data.Analysis — Slice using an index boolean mask
display("Rows 100..104:");
var sliceMask = new PrimitiveDataFrameColumn<bool>("mask", df.Rows.Count);
for(long i = 100; i < 105 && i < df.Rows.Count; i++)
{
    sliceMask[i] = true;
}
df.Filter(sliceMask)

Rows 100..104:

id	symbol	date	open	high	low	close	adj_close	volume	dividends	stock_splits	is_filled
21260	ABI.BR	2021-05-26 00:00:00Z	61.99	62.39	61.83	62.12	58.6701	940186	0.0	0.0	False
21261	ABI.BR	2021-05-27 00:00:00Z	61.8	62.64	61.73	62.13	58.6795	1796477	0.0	0.0	False
21262	ABI.BR	2021-05-28 00:00:00Z	62.14	62.58	61.96	62.34	58.8779	1004125	0.0	0.0	False
21263	ABI.BR	2021-05-31 00:00:00Z	62.27	62.31	61.51	61.56	58.1412	851557	0.0	0.0	False
21264	ABI.BR	2021-06-01 00:00:00Z	62.35	62.48	61.98	62.38	58.9157	1171646	0.0	0.0	False

Microsoft.Data.Analysis | Sort rows with OrderBy and OrderByDescending

MDA provides dataframe-level OrderBy(...) and OrderByDescending(...) for single-column sorting. Chained sorts work, but they are still eager materializations rather than lazy plan transformations.

Sorts the dataframe by close descending to show the top five closing prices, then chains a second sort to inspect symbol-first ordering with descending close values inside the sorted result.

// Microsoft.Data.Analysis — Sort ascending and descending
display("Top 5 by close (descending):");
display(df.OrderByDescending("close").Head(5));
 
// Multi-column sort: Microsoft.Data.Analysis doesn't natively support multi-sort chained via OrderBy().ThenBy()
// We chain OrderBy() calls (which sorts the result of the previous sort).
display("Sort by symbol ASC, then close DESC:");
df.OrderByDescending("close").OrderBy("symbol").Head(5)

Top 5 by close (descending):

id	symbol	date	open	high	low	close	adj_close	volume	dividends	stock_splits	is_filled
3708	RMS.PA	2025-02-14 00:00:00Z	2926.0	2957.0	2813.0	2839.0	2802.9382	105651	0.0	0.0	False
3707	RMS.PA	2025-02-13 00:00:00Z	2770.0	2816.0	2765.0	2816.0	2780.2302	80087	0.0	0.0	False
3709	RMS.PA	2025-02-17 00:00:00Z	2825.0	2858.0	2803.0	2809.0	2776.7424	53852	3.5	0.0	False
3710	RMS.PA	2025-02-18 00:00:00Z	2816.0	2827.0	2780.0	2806.0	2773.7771	65469	0.0	0.0	False
60927	ADYEN.AS	2021-08-24 00:00:00Z	2725.0	2766.0	2711.5	2766.0	2766.0	61431	0.0	0.0	False

Sort by symbol ASC, then close DESC:

id	symbol	date	open	high	low	close	adj_close	volume	dividends	stock_splits	is_filled
21512	ABI.BR	2022-05-17 00:00:00Z	54.65	55.18	53.96	54.44	51.9	1092579	0.0	0.0	False
21814	ABI.BR	2023-07-19 00:00:00Z	51.59	52.15	51.34	52.0	50.2199	1228208	0.0	0.0	False
21408	ABI.BR	2021-12-20 00:00:00Z	51.69	52.32	50.71	52.0	49.1121	2123317	0.0	0.0	False
22457	ABI.BR	2026-01-26 00:00:00Z	59.0	59.44	58.94	59.0	59.0	836059	0.0	0.0	False
21240	ABI.BR	2021-04-28 00:00:00Z	58.09	59.6	58.02	59.0	55.2524	1250347	0.0	0.0	False

Microsoft.Data.Analysis | Distinct counts from ValueCounts

For entity-style dedup checks, the quickest MDA path is usually ValueCounts() or LINQ Distinct() on the key column. That gives you the unique-cardinality view you need even when you do not build a full dataframe-level dedup helper.

Computes ValueCounts() for symbol, prints the number of unique tickers relative to the total row count, and previews the first ten unique symbols with their counts.

// Microsoft.Data.Analysis — Get unique values and counts using ValueCounts
var symbolCounts = df.Columns["symbol"].ValueCounts();
 
display($"Unique symbols: {symbolCounts.Rows.Count} (from {df.Rows.Count} total rows)");
 
// ValueCounts returns a DataFrame with the unique values and their counts
symbolCounts.Head(10)

Unique symbols: 50 (from 66355 total rows)

Values	Counts
ABI.BR	1331
AD.AS	1331
ADS.DE	1324
ADYEN.AS	1331
AI.PA	1331
AIR.PA	1331
ALV.DE	1324
ARGX.BR	1331
ASML.AS	1331
BAS.DE	1324

---

C# Explore, Select and Filter Warnings

MDA Filter requires an explicit boolean column, not an expression

Unlike Polars.NET where df.Filter(Col("x").Gt(5)) works directly, MDA requires constructing a PrimitiveDataFrameColumn<bool> first — more verbose and error-prone.

Column selection by string index in MDA returns a reference, not a copy

df["col"] in MDA returns a reference to the column — mutations propagate to the original DataFrame.

C# Explore, Select and Filter Recommendations

1. Profile before transforming — run Describe(), null counts, and unique counts on every new dataset.
2. Filter early — reduce row count before expensive operations (joins, group_by).
3. Prefer Polars.NET expressions for complex filters — the expression API is more composable and readable than MDA’s manual boolean column construction.
4. Validate column existence before selection — check df.Columns to avoid runtime KeyNotFoundException.

Troubleshooting and failure modes

Symptom	Likely cause	Fix
KeyNotFoundException on column select	Column name not in DataFrame	Check df.Columns or df.Schema before selecting
Filter returns empty DataFrame	Condition too restrictive or type mismatch in comparison	Verify filter values match column dtype
Describe() missing columns	MDA excludes non-numeric columns by default	Handle separately with manual aggregation