04 - awk — Data Processing

23 min read

awk — Data Processing

Quote

“The goal was to see how much of programming we could stuff into one line.”

— Brian Kernighan (co-creator of awk)

awk is the fastest useful tool when the input is line-oriented, the field boundaries are stable, and the output can be emitted in a single pass. This version of the page keeps the original reference density where lookup tables add value, but it cuts away unsupported example volume and replaces it with a smaller set of verified Linux and PowerShell demonstrations.

Linux awk | how it works

These are the mechanics behind every awk program: records, fields, pattern-action rules, and lifecycle blocks.

Linux | awk | record and field model

awk reads one input record at a time. By default a record is one line, and the record is split into fields before the action runs.

These built-ins are the ones you touch most often:

VariableMeaning
$0The entire current record
$1First field
$2Second field
$NFLast field
$(NF-1)Second-to-last field
NRCurrent record number across all inputs
NFNumber of fields in the current record
FNRRecord number within the current input file
FSInput field separator
OFSOutput field separator
RSInput record separator
ORSOutput record separator
FILENAMEName of the current input
OFMTNumeric format used by print
CONVFMTNumber-to-string conversion format

This is the core read model. $0 returns the entire record, while $1 and $2 address individual fields after splitting on whitespace.

echo "alice 42 engineer" | awk '{print $0}'
alice 42 engineer
echo "alice 42 engineer" | awk '{print $1}'
alice
echo "alice 42 engineer" | awk '{print $1,$2}'
alice 42

Access the last field regardless of column count

$NF always resolves to the last field in the current record. That makes it reliable even when earlier columns vary in count.

echo "a b c d e" | awk '{print $NF}'
e
echo "a b c d e" | awk '{print $(NF-1)}'
d

Linux | awk | field separators

The default separator is runs of whitespace. -F or FS lets you switch to explicit delimiters or a regular expression.

Count whitespace-delimited fields

This shows the default split behavior: leading and trailing spaces are ignored, and repeated spaces collapse into one separator.

echo "  a   b   c  " | awk '{print NF}'
3

Set a literal or regex field separator

Use a literal separator when the file format is fixed, and a regex when the same stream can contain more than one delimiter style.

printf 'id,name,amount\n1,alice,42\n' | awk -F',' 'NR==2 {print $2}'
alice
printf 'a,b|c;d\n' | awk -F'[,;|]' '{print $3}'
c

Linux | awk | pattern-action execution

Every awk program is a list of rules. A pattern decides whether the action should run for the current record.

pattern1 { action1 }
pattern2 { action2 }
END      { action3 }

The pattern /ERROR/ runs only for records containing ERROR. Adding NR makes the output immediately actionable when you need to locate the record in a file.

printf 'INFO boot\nERROR disk\nWARN retry\n' | awk '/ERROR/ {print NR, $0}'
2 ERROR disk

Count multiple patterns in one pass

Multiple rules can match the same input stream, so one pass can accumulate several counters before END prints the totals.

printf 'INFO boot\nERROR disk\nWARN retry\nERROR timeout\n' | awk '/ERROR/ {errors++} /WARN/ {warns++} END {print errors, warns}'
2 1

Linux | awk | BEGIN and END blocks

BEGIN is for initialization and headers. END is for summaries and final reporting after all input has been consumed.

Emit a header and a summary

This demonstration prints a CSV header before any rows are processed, accumulates a total from the data rows, and prints the summary in END.

printf 'name,total\nalpha,10\nbeta,15\n' | awk -F',' 'BEGIN {OFS=","; print "name","total"} NR > 1 {total += $2} END {print "grand_total", total}'
name,total
grand_total,25

Linux awk | field extraction and formatting

This section covers the most common data-engineering tasks: choosing columns, reordering them, and controlling output layout.

Linux | awk | selecting and reordering columns

These patterns keep input parsing simple while making the output shape explicit.

Field addresses work the same whether the input is whitespace-delimited or separated by an explicit delimiter.

printf 'alice 42 engineer\nbob 37 analyst\n' | awk '{print $1, $3}'
alice engineer
bob analyst
printf 'id,name,amount\n1,alice,42\n2,bob,55\n' | awk -F',' 'NR>1 {print $2}'
alice
bob

Reorder CSV columns

Awk does not care about original column order once the record is split. Reordering is just a different print list.

printf 'name,date,amount\nalice,2026-03-22,42\n' | awk -F',' -v OFS=',' 'NR>1 {print $2,$3,$1}'
2026-03-22,42,alice

Extract a field range

Field ranges are built with a loop. This pattern is useful when you need a suffix of wide records without enumerating every field manually.

printf 'a b c d e f\n' | awk '{for(i=2;i<=5;i++) printf "%s%s",$i,(i<5?OFS:ORS)}'
b c d e

Linux | awk | output formatting

Formatting is a separate decision from input parsing. OFS controls joined output, while printf controls exact layout.

Rebuild $0 with a new output separator

Setting OFS alone does not change $0. Assigning a field forces awk to reconstruct the record using the new separator.

printf 'id,name,amount\n1,alice,42\n' | awk -F',' -v OFS='|' 'NR>1 {$1=$1; print}'
1|alice|42

Format aligned reports with printf

Use printf when alignment and numeric precision matter more than raw delimiter conversion.

printf 'alice 42.135\nbob 7.5\n' | awk '{printf "%-10s %8.2f\n", $1, $2}'
alice         42.13
bob            7.50

Quote fields in generated CSV-like output

This pattern is useful when awk is generating rows for a downstream tool and you need exact punctuation rather than OFS-joined fields.

printf 'alice,42,engineer\n' | awk -F',' '{printf "\"%s\",%s,\"%s\"\n", $1, $2, $3}'
"alice",42,"engineer"

Linux | awk | printf format specifiers

These are the format codes you will use most often in reporting-style output.

SpecifierMeaning
%sString
%dInteger
%fFloating-point number
%eScientific notation
%gShorter of %f or %e
%-20sLeft-aligned, width 20
%10.2fWidth 10, 2 decimal places
%08dZero-padded integer
`{init: {‘theme’: ‘dark’, ‘themeVariables’: {‘primaryColor’: ‘#292e42’,‘primaryTextColor’: ‘#c0caf5’,‘primaryBorderColor’: ‘#565f89’,‘lineColor’: ‘#565f89’,‘secondaryColor’: ‘#1a1b26’,‘tertiaryColor’: ‘#24283b’,‘noteTextColor’: ‘#c0caf5’,‘noteBkgColor’: ‘#292e42’,‘textColor’: ‘#c0caf5’,‘fontSize’: ‘14px’}}}%%
flowchart TD
A([awk invoked]) --> B[Execute BEGIN block]
B --> C{More input records?}
C -- yes --> D[Read next record into $0]
D --> E[Split $0 into fields<br>$1 $2 ... $NF]
E --> F{Evaluate each<br>pattern-action rule}
F -- pattern matches --> G[Execute action]
F -- no match / default --> H[Next rule]
G --> H
H --> F
F -- all rules evaluated --> C
C -- no --> I[Execute END block]
I --> J([Output complete])

style A fill:#292e42,stroke:#565f89
style J fill:#292e42,stroke:#565f89
style B fill:#1a1b26,stroke:#565f89
style I fill:#1a1b26,stroke:#565f89
style D fill:#24283b,stroke:#565f89
style E fill:#24283b,stroke:#565f89
style G fill:#24283b,stroke:#565f89


## PowerShell awk equivalents

PowerShell passes structured objects instead of text records, so the direct awk translation is often "parse once, then address named properties." For inline demonstrations below, `ConvertFrom-Csv` stands in for file-backed `Import-Csv`.

### PowerShell | column selection

These are the closest equivalents to awk field projection when the input is already CSV-shaped.

#### Select named columns from CSV objects

This is the Windows-native equivalent of "split the row once, then print only the fields you care about."

```powershell
@"
id,name,amount
1,alice,42
2,bob,55
"@ | ConvertFrom-Csv | ForEach-Object { "$($_.id),$($_.amount)" }

1,42
2,55

Add headers when raw rows have no header line

This mirrors positional field extraction when the source data lacks names.

@"
1,alice,42
2,bob,55
"@ | ConvertFrom-Csv -Header id,name,amount | ForEach-Object { "$($_.id):$($_.amount)" }
1:42
2:55

PowerShell | filtering and aggregation

Where-Object and Measure-Object cover most of the filtering and summary work that awk handles with record tests and accumulators.

Filter rows by numeric threshold

PowerShell makes the numeric conversion explicit, which is the same discipline awk needs with +0.

@"
name,amount,status
alice,42,OK
bob,105,FAIL
cara,70,OK
"@ | ConvertFrom-Csv | Where-Object { [int]$_.amount -gt 50 } | ForEach-Object { "$($_.name) $($_.amount)" }
bob 105
cara 70

Count matching rows and summarize numeric columns

The first pipeline counts records meeting a predicate. The second computes summary statistics across a numeric property.

@"
name,amount,status
alice,42,ACTIVE
bob,105,FAIL
cara,70,ACTIVE
"@ | ConvertFrom-Csv | Where-Object { $_.status -eq 'ACTIVE' } | Measure-Object | Select-Object -ExpandProperty Count
2
@"
name,amount
alice,42
bob,58
"@ | ConvertFrom-Csv | Measure-Object -Property amount -Sum -Average | ForEach-Object { "sum=$([int]$_.Sum) avg=$([math]::Round($_.Average, 2))" }
sum=100 avg=50

PowerShell | grouping and shaping

Grouping, calculated properties, and projection are where the PowerShell object pipeline becomes clearer than manual text splitting.

Group by category and sum amount

This is the PowerShell equivalent of sum[$1]+=$2 followed by an END block.

@"
category,amount
retail,10
finance,20
retail,5
"@ | ConvertFrom-Csv | Group-Object category | Sort-Object Name | ForEach-Object { "$($_.Name) $(($_.Group | Measure-Object amount -Sum).Sum)" }
finance 20
retail 15

Add uppercase or calculated properties

Calculated properties are the object-pipeline replacement for awk expressions embedded in print or printf.

@"
id,name
1,alice
2,bob
"@ | ConvertFrom-Csv | Select-Object id, @{Name='name';Expression={$_.name.ToUpper()}} | ForEach-Object { "$($_.id) $($_.name)" }
1 ALICE
2 BOB
@"
name,revenue,cost
alpha,100,70
beta,80,20
"@ | ConvertFrom-Csv | Select-Object name, @{Name='margin_pct';Expression={ [math]::Round((([double]$_.revenue - [double]$_.cost) / [double]$_.revenue) * 100, 2) }} | ForEach-Object { '{0} {1:N2}' -f $_.name, $_.margin_pct }
alpha 30.00
beta 75.00

PowerShell | text-oriented fallbacks

When the input is raw text rather than structured objects, PowerShell can still handle the job without delegating back to awk.

Remove duplicate lines from raw text

This is the closest equivalent to !seen[$0]++ on a plain text stream.

@"
alpha
beta
alpha
"@ -split "`n" | Where-Object { $_ } | Select-Object -Unique
alpha
beta

Emit tab-delimited text without reparsing in awk

For inline transforms, a formatted string is often simpler than writing an intermediate file and re-importing it.

@"
id,name,amount
1,alice,42
2,bob,55
"@ | ConvertFrom-Csv | ForEach-Object { "$($_.id)`t$($_.name)`t$($_.amount)" }
1	alice	42
2	bob	55

PowerShell | quick pattern equivalents

These examples map a few common awk one-liners onto idiomatic PowerShell.

This is the object-pipeline version of NR % 5 == 0.

1..10 | ForEach-Object { if($_ % 5 -eq 0) { $_ } }
5
10

When the input is still plain text, split the line and read the last element of the resulting array.

(@"
alpha beta gamma
one two three
"@ -split "`n") | Where-Object { $_ } | ForEach-Object { ($_ -split '\s+')[-1] }
gamma
three

PowerShell | comparison table: awk vs PowerShell

Use this table as a quick translator between the awk mindset and the PowerShell object pipeline.

TaskawkPowerShell
Parse CSVawk -F','Import-Csv / ConvertFrom-Csv
Filter rows$3 > 100 {print}Where-Object { [int]$_.col -gt 100 }
Select columns{print $1,$3}Select-Object col1, col3
Count rowsEND {print NR}.Count / Measure-Object
Sum a column{sum+=$3} END{print sum}Measure-Object -Sum
Group-byAssociative arrayGroup-Object
Add a calculated column{print $1, $2*$3}Select-Object @{Name=...; Expression={...}}
Deduplicate rows!seen[$0]++Select-Object -Unique
Replace textgsub(/x/,"y")-replace 'x','y'
Uppercasetoupper($1)$_.col.ToUpper()
Convert delimitersawk -F',' -v OFS='\t' '{$1=$1; print}'ForEach-Object { "...t…” }/Export-Csv -Delimiter`
Every Nth rowNR%100==0 {print}ForEach-Object { if(...) { ... } }
Join two filesFNR==NR lookup trickGroup-Object, hash table, or custom lookup
Write to a fileprint > "out.txt"Set-Content / Out-File

Linux awk | quick reference card

This section keeps a compact set of self-contained one-liners, but each item is still demonstrated and verified.

Linux | awk | verified one-liners

Each item below is safe to paste into a shell when you need a quick reminder.

Number every line

Prefixing output with NR is the fastest debugging move when you need positional context.

printf 'alpha\nbeta\n' | awk '{print NR": "$0}'
1: alpha
2: beta

Remove blank lines

NF > 0 is the simplest predicate for keeping only non-empty records.

printf 'alpha\n\nbeta\n' | awk 'NF > 0'
alpha
beta

This is the “show me second and later sightings” pattern from the duplicates section, kept here because it is worth memorizing.

printf 'alpha\nbeta\nalpha\nalpha\n' | awk 'seen[$0]++ > 0'
alpha
alpha

Sum a single-column file

For one numeric column, the accumulator can be expressed in one short rule and one summary block.

printf '10\n15\n5\n' | awk '{s+=$1} END{print s}'
30

Validate a fixed field count

This pattern is useful in ETL checks where malformed rows must be surfaced before a load runs.

printf 'a,b,c,d,e\n1,2,3\n' | awk -F',' 'NF != 5 {print "BAD ROW:", NR, NF, $0}'
BAD ROW: 2 3 1,2,3

Tracking the seen key instead of the whole row is the right pattern when uniqueness depends on one column only.

printf '1,alice\n2,bob\n3,alice\n' | awk -F',' '!seen[$2]++ {print $2}'
alice
bob

When to use awk vs sed

sed and awk overlap on regex matching, but they are optimized for different jobs. sed is a stream editor. awk is a field-aware programming language.

Linux | tool choice | prefer sed

Choose sed when the job is fundamentally line editing rather than field-aware transformation.

Line-oriented substitutions

For pure substitution, sed is shorter and clearer. Awk can do the same job, but the extra machinery is unnecessary unless you also need fields or state.

printf 'alpha beta\n' | sed 's/a/A/g'
AlphA betA
printf 'alpha beta\n' | awk '{gsub(/a/, "A"); print}'
AlphA betA

In-place file edits

In-place editing is a core sed use case. Awk can rewrite files, but it does not have a native equivalent to sed -i.

tmpfile=$(mktemp)
printf 'alpha\n' > "$tmpfile"
sed -i 's/alpha/ALPHA/' "$tmpfile"
cat "$tmpfile"
rm -f "$tmpfile"
ALPHA

Linux | tool choice | prefer awk

Choose awk when the record must be split into fields or when the result depends on arithmetic or state across records.

Field-aware extraction

This is the category of work sed does not model well at all.

printf '1,alice,42\n' | awk -F',' '{print $2, $3}'
alice 42

Arithmetic and aggregation

Once the job needs numeric accumulation or grouping, awk is the right shell-native tool.

printf 'retail,10\nfinance,20\nretail,5\n' | awk -F',' '{sum[$1]+=$2} END {for (k in sum) print k, sum[k]}' | sort
finance 20
retail 15

Multi-file lookups

Associative arrays plus FNR==NR make cross-file enrichment practical without leaving the shell.

awk -F',' 'FNR==NR {name[$1]=$2; next} {print $1, name[$1], $2}' <(printf '1,alice\n2,bob\n') <(printf '1,42\n2,55\n')
1 alice 42
2 bob 55

Formatted reports

printf is where awk starts looking like a compact reporting language rather than a simple filter.

printf 'alice 42.135\nbob 7.5\n' | awk '{printf "%-10s %8.2f\n", $1, $2}'
alice         42.13
bob            7.50

Linux | tool choice | escalate beyond both

Some text-processing tasks are not good fits for either sed or plain awk.

Quoted CSV or nested structures

This broken parse is the signal to switch tools. Plain -F',' has no notion of quoted commas inside a field.

printf '"Smith, John",42\n' | awk -F',' '{print $1 "|" $2}'
"Smith| John"

For real CSV, use a CSV-aware parser such as mlr, Python’s csv module, or PowerShell’s CSV cmdlets.

awk Data Processing Recommendations

These are the safest default patterns for common awk tasks. The H4 titles mirror the original table entries so the decision logic stays visible.

Linux | recommendations by scenario

Use these as starting templates, then specialize the predicate or printed fields.

Extract specific columns

Set -F to the actual delimiter and print only the fields you need.

printf '1,alice,42\n2,bob,55\n' | awk -F',' '{print $1, $3}'
1 42
2 55

Skip the header row

NR > 1 is the standard guard when the first line contains column names rather than data.

printf 'id,name,amount\n1,alice,42\n2,bob,55\n' | awk -F',' 'NR>1 {print $2}'
alice
bob

Sum a numeric column

Convert implicitly numeric fields with arithmetic and emit the total in END.

printf 'name,amount\nalpha,10\nbeta,15\n' | awk -F',' 'NR>1 {sum += $2} END {print sum}'
25

Count unique values

Track first sightings with an associative array and count the distinct keys.

printf 'retail\nfinance\nretail\n' | awk '!seen[$0]++ {count++} END {print count}'
2

Filter by field value

Make the comparison type explicit when the field is numeric.

printf 'name,amount,status\nalice,42,OK\nbob,105,FAIL\ncara,70,OK\n' | awk -F',' 'NR>1 && $2+0 > 50 {print $1, $2}'
bob 105
cara 70

Validate record structure

NF is the first integrity check to run against delimiter-separated data before a load or downstream transformation.

printf 'a,b,c,d,e\n1,2,3\n' | awk -F',' 'NF != 5 {print "BAD ROW:", NR, NF, $0}'
BAD ROW: 2 3 1,2,3

Produce CSV output

Set OFS="," and let print rebuild the row with explicit comma separators.

printf 'alice 42 engineer\n' | awk 'BEGIN{OFS=","} {print $1,$2,$3}'
alice,42,engineer

PowerShell | recommendations by scenario

On Windows, prefer object-aware CSV parsing over manual string splitting whenever the data already has headers.

Use ConvertFrom-Csv or Import-Csv for named columns

This is the direct replacement for positional CSV extraction when column names are available.

@"
id,name,amount
1,alice,42
2,bob,55
"@ | ConvertFrom-Csv | ForEach-Object { "$($_.id),$($_.amount)" }
1,42
2,55

awk Data Processing Troubleshooting

These are the failure modes that show up most often when awk scripts are moved from toy data to production-like input.

Linux | awk troubleshooting | parsing and field boundaries

Start by proving what awk thinks the fields are. Most failures in this category come from an incorrect parse model.

Fields are split incorrectly

If you forget -F',', awk treats the entire CSV row as one whitespace-delimited field. Adding the correct separator fixes the field count immediately.

printf '1,alice,42\n' | awk '{print NF, $1}'
1 1,alice,42
printf '1,alice,42\n' | awk -F',' '{print NF, $1, $2, $3}'
3 1 alice 42

Quoted CSV fields are corrupted

Plain field splitting breaks as soon as a quoted field contains the delimiter.

printf '"Smith, John",42\n' | awk -F',' '{print $1 "|" $2}'
"Smith| John"

When this happens, switch to a CSV-aware parser instead of trying to patch plain awk field splitting.

Numeric comparisons behave like strings

If the input field is still a string, string comparison rules apply. Force numeric coercion with +0 before comparing.

printf '9\n10\n' | awk '{print $1, ($1 > "9" ? "string-gt-9" : "string-not-gt-9")}'
9 string-not-gt-9
10 string-not-gt-9
printf '9\n10\n' | awk '{print $1, ($1+0 > 9 ? "number-gt-9" : "number-not-gt-9")}'
9 number-not-gt-9
10 number-gt-9

Linux | awk troubleshooting | output and control-flow surprises

Once parsing is correct, the next failures are usually formatting and empty-input edge cases.

printf output appears on one line

printf writes exactly what the format string says. Without \n, separate records concatenate together.

printf 'alpha 1\nbeta 2\n' | awk '{printf "%s:%s", $1, $2}'
alpha:1beta:2
printf 'alpha 1\nbeta 2\n' | awk '{printf "%s:%s\n", $1, $2}'
alpha:1
beta:2

END logic runs with no data

END always runs, even if the input is empty, so guard your summary logic when zero-row input is possible.

printf '' | awk 'END {print (NR==0 ? "no input" : NR)}'
no input

The workflow no longer fits a one-pass awk script

If the script now needs full CSV quoting, deep nesting, multi-pass joins, or nontrivial data structures, stop forcing awk to be a general-purpose language. Rewrite the workflow in Python, SQL, or a structured ETL tool before the script becomes impossible to reason about.

Linux awk | arithmetic and aggregation

This section covers the numeric helpers, string transforms, and accumulator patterns that turn awk into a compact data-processing language.

Linux | awk | built-in arithmetic functions

These helpers cover most lightweight numeric work in awk.

FunctionDescription
int(x)Truncate x toward zero
sqrt(x)Square root
exp(x)Natural exponential
log(x)Natural logarithm
sin(x)Sine in radians
cos(x)Cosine in radians
atan2(y, x)Arctangent of y/x
rand()Random float between 0 and 1
srand(seed)Seed the random generator

Linux | awk | arithmetic and string transformation

These patterns cover the most common numeric and string reshaping tasks in data pipelines.

Compute a derived metric

Derived fields are often the point where awk replaces a throwaway spreadsheet step.

printf 'name,revenue,cost\nalpha,100,70\nbeta,80,20\n' | awk -F',' 'NR>1 {margin=($2-$3)/$2*100; printf "%s %.2f\n", $1, margin}'
alpha 30.00
beta 75.00

Truncate floating-point values with int()

int() truncates toward zero, which is often what you want for bucket calculations and whole-number summaries.

echo "3.7" | awk '{print int($1)}'
3

Replace the first or all matching substrings

sub() changes only the first match, while gsub() replaces every match in the target string.

printf 'ERROR disk,ERROR retry\n' | awk '{sub(/ERROR/, "WARN", $0); print}'
WARN disk,ERROR retry
printf 'data engineer\n' | awk '{gsub(/ /, "_", $0); print}'
data_engineer

Use GNU-only gensub() when backreferences matter

gensub() is a gawk extension. Use it when the replacement needs captured groups or when you need to target a specific occurrence.

echo "2026-03-22" | gawk '{print gensub(/([0-9]{4})-([0-9]{2})-([0-9]{2})/, "\\3/\\2/\\1", "g")}'
22/03/2026
echo "foo_bar_baz" | gawk '{print gensub(/_/, "-", 2)}'
foo_bar-baz

Capture values with match()

Use the GNU array form when you need captured groups, and the POSIX form when you only need the matching slice.

echo "error code=42 msg=timeout" | gawk '{
    match($0, /code=([0-9]+) msg=([a-z]+)/, arr)
    print "Code:", arr[1], "Message:", arr[2]
}'
Code: 42 Message: timeout
echo "error code=42" | awk '{
    if (match($0, /code=[0-9]+/))
        print substr($0, RSTART, RLENGTH)
}'
code=42

Linux | awk | aggregation patterns

Associative arrays and running totals are the features that make awk useful far beyond simple field projection.

Sum a column

This is the standard one-pass accumulator pattern for numeric totals.

printf 'name,amount\nalpha,10\nbeta,15\n' | awk -F',' 'NR>1 {sum += $2} END {print sum}'
25

Compute multiple statistics in one pass

You can collect count, sum, average, minimum, and maximum in one scan without leaving awk.

printf '10\n15\n5\n' | awk 'NR==1{min=max=$1} {sum+=$1; count++; if($1<min) min=$1; if($1>max) max=$1} END {printf "count=%d sum=%d avg=%.2f min=%d max=%d\n", count, sum, sum/count, min, max}'
count=3 sum=30 avg=10.00 min=5 max=15

Group and total by key

This is the awk equivalent of GROUP BY category SUM(amount).

printf 'retail,10\nfinance,20\nretail,5\n' | awk -F',' '{sum[$1]+=$2} END {for (k in sum) print k, sum[k]}' | sort
finance 20
retail 15

Carry a running total through the stream

Running totals are useful when you need cumulative output instead of a single summary line at the end.

printf '10\n15\n5\n' | awk '{sum+=$1; print NR, sum}'
1 10
2 25
3 30

Linux awk | data engineering scenarios

These are representative tasks where awk is still a good fit in production-oriented shell workflows.

Linux | awk | simple CSV cleanup

These examples assume uncomplicated delimiter-separated data without quoted commas.

Trim surrounding whitespace from every field

This pattern normalizes a messy CSV export before a downstream load step.

printf 'id,name,amount\n1, alice ,42\n2, bob ,55\n' | awk 'BEGIN{FS=","; OFS=","} NR==1{print; next} {for(i=1;i<=NF;i++) gsub(/^[[:space:]]+|[[:space:]]+$/, "", $i); print}'
id,name,amount
1,alice,42
2,bob,55

Replace empty fields with a placeholder

This keeps record width stable when downstream consumers need an explicit null marker.

printf 'id,name,amount\n1,alice,\n2,,55\n' | awk 'BEGIN{FS=","; OFS=","} {for(i=1;i<=NF;i++) if($i=="") $i="NULL"; print}'
id,name,amount
1,alice,NULL
2,NULL,55

Linux | awk | pipeline-log summaries

Timestamped logs are a good fit for one-pass aggregation when the date is already present in each record.

Count rows per day

This extracts the date prefix from the timestamp and increments an associative-array counter per day.

printf '2026-03-22T10:00:00Z pipeline=ingest status=OK\n2026-03-22T11:00:00Z pipeline=ingest status=FAIL\n2026-03-23T09:30:00Z pipeline=sync status=OK\n' | awk '{day=substr($1,1,10); count[day]++} END {for (d in count) print d, count[d]}' | sort
2026-03-22 2
2026-03-23 1

Count failures per day

Adding a status filter turns the same pattern into a daily failure summary.

printf '2026-03-22T10:00:00Z pipeline=ingest status=OK\n2026-03-22T11:00:00Z pipeline=ingest status=FAIL\n2026-03-23T09:30:00Z pipeline=sync status=FAIL\n' | awk '/status=FAIL/ {day=substr($1,1,10); fail[day]++} END {for (d in fail) print d, fail[d]}' | sort
2026-03-22 1
2026-03-23 1

Linux | awk | key=value logs

Key-value records are common in application logs and batch status output.

Extract one key from each record

This loops over fields and selects only the user= token.

printf 'ts=2026-03-22 level=INFO user=alice\nts=2026-03-22 level=ERROR user=bob\n' | awk '{for(i=1;i<=NF;i++) if($i ~ /^user=/) {split($i,a,"="); print a[2]}}'
alice
bob

Build a map for later field access

Once the line is normalized into an associative array, you can access the keys by name rather than by original position.

printf 'ts=2026-03-22 level=ERROR user=bob retries=3\n' | awk '{for(i=1;i<=NF;i++){split($i,a,"="); kv[a[1]]=a[2]} print kv["level"], kv["user"], kv["retries"]}'
ERROR bob 3

Linux | awk | duplicate detection

Associative arrays make deduplication and frequency counts straightforward.

This prints the second and later occurrences while suppressing the first sighting of each record.

printf 'alpha\nbeta\nalpha\nalpha\n' | awk 'seen[$0]++ > 0'
alpha
alpha

Count occurrences per unique line

This is the simplest frequency-table pattern in awk.

printf 'alpha\nbeta\nalpha\nalpha\n' | awk '{count[$0]++} END {for (k in count) print k, count[k]}' | sort
alpha 3
beta 1

Deduplicate by key column

When the whole row can change but the key column is authoritative, track the first-seen key instead of the whole line.

printf '1,alice\n2,bob\n1,alice-new\n' | awk -F',' '!seen[$1]++ {print $0}'
1,alice
2,bob

Linux | awk | line-ending cleanup

CRLF cleanup is a small but frequent interoperability task when Windows-generated text lands in Unix pipelines.

Strip carriage returns from CRLF input

Removing \r normalizes the stream so later field handling behaves predictably.

printf 'alpha\r\nbeta\r\n' | awk '{gsub(/\r/, ""); print}'
alpha
beta

awk Data Processing Cross-References

Graph View

Backlinks