Process Substitution and Here Documents — Advanced Input/Output

Quote

“Expect the output of every program to become the input to another, as yet unknown, program. Don’t clutter output with extraneous information. Don’t insist on interactive input.”

Doug McIlroy, Bell System Technical Journal (1978)

These features let you route generated data through file-oriented tools and stdin-driven commands without dropping intermediate files into your working directory.

Linux process substitution tools

Process substitution creates a virtual file descriptor that wraps a command’s output so other commands can read from it as if it were a file. No temporary files are created or cleaned up. The kernel provides the file descriptor transparently via /dev/fd/N (or a named pipe on systems that lack /dev/fd).

Two forms exist: <(cmd) produces a readable file descriptor (input substitution), and >(cmd) produces a writable file descriptor (output substitution). Both can appear on the same command line.


flowchart LR
    subgraph ps_in["Input substitution  <(cmd)"]
        A[cmd] -->|stdout| FD1["/dev/fd/N<br>virtual file"]
        FD1 -->|read| B[consumer command]
    end
    subgraph ps_out["Output substitution  >(cmd)"]
        C[producer command] -->|write| FD2["/dev/fd/M<br>virtual file"]
        FD2 -->|stdin| D[cmd]
    end

Linux | process substitution | input <()

Input substitution replaces a filename argument with a live file descriptor backed by the output of the given command. The outer command reads from that descriptor exactly as it would read from a real file.

Linux | <() | compare two command outputs with diff

diff requires two file paths. Using <() lets you feed it two in-memory streams without creating any disk files. Each <(sort ...) expression becomes a separate /dev/fd/N descriptor that diff reads sequentially.

diff <(sort file1.txt) <(sort file2.txt)
2c2
< banana
---
> cherry

Linux | <() | compare row counts without temp files

Use the same pattern when two generated counts need to be compared by a file-oriented tool. This safe local example feeds diff two synthetic counts; the same structure works with database clients once you have a real query runner.

diff <(printf '%s\n' 2451) <(printf '%s\n' 2449)
1c1
< 2451
---
> 2449

Linux | <() | merge selected columns with paste

paste joins lines from multiple files side by side. Wrapping each cut call in <() extracts two non-adjacent columns from the same CSV and merges them into a new stream without writing an intermediate file.

paste <(cut -d, -f1 stocks.csv) <(cut -d, -f3 stocks.csv)
AAPL    145.32
MSFT    310.05
ASML    720.00

Linux | process substitution | output >()

Output substitution creates a writable file descriptor backed by the given command’s stdin. The producer writes to the substitution as if writing to a file, and the backing command receives that data on its own stdin.

Linux | >() | tee output to multiple simultaneous consumers

tee copies its stdin to each destination listed. Using >() lets each destination be a live process rather than a file, so compression and line-counting happen concurrently from one read of the input stream. The follow-up reads back both generated artifacts.

tmpdir=$(mktemp -d)
printf '%s\n' alpha beta gamma | tee >(gzip > "$tmpdir/data.gz") >(wc -l > "$tmpdir/count.txt") > /dev/null
printf 'count=%s\n' "$(cat "$tmpdir/count.txt")"
gzip -cd "$tmpdir/data.gz"
rm -rf "$tmpdir"
count=3
alpha
beta
gamma

The redirect > /dev/null suppresses the copy that tee would normally write to stdout, since both useful outputs are handled by the two substitutions.

Linux | >() | log and process in parallel

Writing to two independent >(...) sinks lets you archive raw data to a log file while simultaneously processing it through a filter, all in one pipeline pass. The verification commands read both outputs back so the split is visible.

tmpdir=$(mktemp -d)
printf '%s\n' 'INFO started' 'ERROR disk full' 'INFO retry' | tee >(cat > "$tmpdir/audit.log") >(grep ERROR > "$tmpdir/errors.txt") > /dev/null
printf 'audit.log:\n'
cat "$tmpdir/audit.log"
printf 'errors.txt:\n'
cat "$tmpdir/errors.txt"
rm -rf "$tmpdir"
audit.log:
INFO started
ERROR disk full
INFO retry
errors.txt:
ERROR disk full

Linux | process substitution | flag reference

Use the table as a quick syntax lookup after the examples above establish the data flow.

SyntaxDescription
<(cmd)Run cmd and expose its stdout as a readable file descriptor
>(cmd)Expose a writable file descriptor; data written to it reaches cmd’s stdin
/dev/fd/NThe actual path the shell provides for the substitution

Linux here document tools

A here document embeds multi-line text directly in a script, feeding it as stdin to a command. The shell reads lines until it encounters the closing delimiter on its own line. Here documents are useful for inline SQL, config generation, and any command that reads a block of input.

Linux | here document | literal << 'EOF'

Enclosing the opening delimiter in single quotes (<< 'EOF') disables all expansion inside the block: variables, command substitutions, and backslash escapes are passed through literally. Use this form when you want the exact text without any interpretation.

Linux | << 'EOF' | preserve literal text with dollar signs

Use a quoted delimiter when the consumer must receive the text unchanged. This local example prints SQL-like text verbatim; the same form is what you want before piping into sqlcmd or another client that interprets $... itself.

cat <<'EOF'
SELECT '$USER' AS literal_user;
EOF
SELECT '$USER' AS literal_user;

Linux | here document | expanding << EOF

When the opening delimiter is unquoted (<< EOF), the shell expands $variables, $(command) substitutions, and backslash sequences inside the block before passing it to the command. Use this form for dynamic content generation.

Linux | << EOF | generate a config file with variable expansion

Variables and command substitutions in the block are resolved at the time the script runs. The final output is redirected into config.env, creating or overwriting the file. The verification step reads the generated file back so the expansion is visible.

tmpdir=$(mktemp -d)
DB_HOST=db.internal
DB_PORT=5432
cat << EOF > "$tmpdir/config.env"
DB_HOST=${DB_HOST}
DB_PORT=${DB_PORT}
PIPELINE_NAME=pipeline_daily
GENERATED_BY=$(printf 'bash-demo')
EOF
cat "$tmpdir/config.env"
rm -rf "$tmpdir"
DB_HOST=db.internal
DB_PORT=5432
PIPELINE_NAME=pipeline_daily
GENERATED_BY=bash-demo

Linux | << EOF | feed multiple commands to another shell

A here document can drive any stdin-reading program, not just cat. This local example sends two commands to a nested Bash process; the same pattern is how multi-line ssh user@host <<EOF sessions work once a remote host is available.

bash <<'EOF'
cd /tmp
pwd
printf '%s\n' ready
EOF
/tmp
ready

Linux | here document | flag reference

Keep the table for the small syntax differences; the earlier examples carry the behavior.

SyntaxDescription
<< 'EOF'Literal here document — no variable or command expansion
<< EOFExpanding here document — $var and $(cmd) are substituted
<<- EOFExpanding here document; leading tabs (not spaces) are stripped from each line
<< 'EOF' > fileRedirect here document output to a file

Linux here string tools

A here string feeds one short string to stdin without a temporary file or a separate printf | pipeline. It is the cleanest way to pass a literal value on stdin.

Linux | here string | <<<

Linux | <<< | grep a literal string

The string "ASML SAP SIE" is fed directly to grep’s stdin. This avoids a pipe and an echo subprocess, and the intent is immediately clear from the <<< syntax.

grep "ASML" <<< "ASML SAP SIE"
ASML SAP SIE

Linux | <<< | parse a variable with read

read normally reads from the terminal. Redirecting a here string into it assigns substrings to named variables without a pipe or subshell.

read first rest <<< "AAPL 145.32 2026-01-15"
echo "$first"
echo "$rest"
AAPL
145.32 2026-01-15

Linux | <<< | feed a JSON string to jq

Here strings work with any command that reads from stdin. Passing a JSON literal directly via <<< is cleaner than wrapping it in echo or writing a file.

jq '.price' <<< '{"symbol":"AAPL","price":145.32}'
145.32

Linux | here string | flag reference

Use the table as a syntax reminder once you know that <<< is for short stdin payloads, not large multi-line blocks.

SyntaxDescription
<<< "string"Feed a literal string as stdin; variables are expanded
<<< '$literal'Single-quoted string; no expansion
<<< "$(cmd)"Feed command output as a single-line stdin

PowerShell process substitution tools

PowerShell has no direct equivalent to Bash process substitution (<(), >()). Instead, use temporary files when a command needs a path, Tee-Object -Variable when one in-memory capture is enough, and ForEach-Object when you need custom branching logic.

PowerShell | process substitution workaround | temporary files

The most direct translation of <(cmd) is to write command output to a temporary file, use the file, then delete it. This is verbose but always works and is easy to audit.

PowerShell | temp file | compare two command outputs with diff

Write each command’s output to a temporary file, run Compare-Object (PowerShell’s diff equivalent), then clean up. [System.IO.Path]::GetTempFileName() returns a unique path in the system temp directory.

$f1 = [System.IO.Path]::GetTempFileName()
$f2 = [System.IO.Path]::GetTempFileName()
(Get-Content file1.txt | Sort-Object) | Set-Content $f1
(Get-Content file2.txt | Sort-Object) | Set-Content $f2
Compare-Object (Get-Content $f1) (Get-Content $f2)
Remove-Item $f1, $f2
InputObject SideIndicator
----------- -------------
cherry      =>
banana      <=

PowerShell | process substitution workaround | pipeline variables

Tee-Object -Variable captures pipeline output into a named variable while still passing it downstream. This is the closest equivalent to tee >(cmd) for in-memory branching.

PowerShell | Tee-Object | capture and pass through simultaneously

The pipeline output is both stored in $lines and forwarded to Measure-Object. No file is written; both operations happen in a single pipeline.

Get-Content data.csv |
    Tee-Object -Variable lines |
    Measure-Object -Line
Lines Words Characters Property
----- ----- ---------- --------
  500

PowerShell | process substitution workaround | ForEach-Object branching

For multiple simultaneous consumers, ForEach-Object with a Begin/Process/End script block or parallel calls can approximate tee >() >() by dispatching each item to multiple operations inline.

PowerShell | ForEach-Object | send output to two destinations

Each line from the pipeline is appended to audit.log and simultaneously tested for the string ERROR. Matches are collected in $errors, and the verification step reads both outputs back from a temp directory.

$tmp = Join-Path $env:TEMP ('ps-sub-' + [guid]::NewGuid())
New-Item -ItemType Directory -Path $tmp | Out-Null
@('INFO started','ERROR disk full','INFO retry') | Set-Content (Join-Path $tmp 'pipeline.log')
$errors = [System.Collections.Generic.List[string]]::new()
Get-Content (Join-Path $tmp 'pipeline.log') | ForEach-Object {
    Add-Content -Path (Join-Path $tmp 'audit.log') -Value $_
    if ($_ -match 'ERROR') { $errors.Add($_) }
}
$errors | Set-Content (Join-Path $tmp 'errors.txt')
'audit.log:'
Get-Content (Join-Path $tmp 'audit.log')
'errors.txt:'
Get-Content (Join-Path $tmp 'errors.txt')
Remove-Item -LiteralPath $tmp -Recurse -Force
audit.log:
INFO started
ERROR disk full
INFO retry
errors.txt:
ERROR disk full

PowerShell here document tools

PowerShell’s equivalent of a here document is the here-string, written with @'...'@ (literal) or @"..."@ (expanding). Unlike Bash, the opening @' or @" must sit at the end of the line, and the closing '@ or "@ must appear at the start of a line with no leading whitespace.

PowerShell | here-string | literal @'...'@

PowerShell | @'...'@ | preserve a literal SQL block

Everything between @' and '@ is treated as a literal string. No variable or expression expansion occurs. This local example keeps $releaseTag literal inside a SQL-like block; the same pattern is what you want before passing the text to Invoke-Sqlcmd.

$releaseTag = 'v2.0'
$query = @'
SELECT '$releaseTag' AS literal_tag;
'@
$query
SELECT '$releaseTag' AS literal_tag;

PowerShell | here-string | expanding @"..."@

PowerShell | @"..."@ | generate a config block with variable expansion

Variables and subexpressions inside @"..."@ are expanded before the string is used. The result is written to config.env, and the verification step reads the generated file back.

$tmp = Join-Path $env:TEMP ('ps-here-' + [guid]::NewGuid() + '.env')
$env:DB_HOST = 'db.internal'
$env:DB_PORT = '5432'
$content = @"
DB_HOST=$env:DB_HOST
DB_PORT=$env:DB_PORT
PIPELINE_NAME=pipeline_daily
GENERATED_BY=$( '2026-04-14T00:00:00Z' )
"@
Set-Content -Path $tmp -Value $content
Get-Content $tmp
Remove-Item -LiteralPath $tmp -Force
DB_HOST=db.internal
DB_PORT=5432
PIPELINE_NAME=pipeline_daily
GENERATED_BY=2026-04-14T00:00:00Z

PowerShell | here-string | flag reference

Keep the table as a compact syntax check; the examples above show when each form is appropriate.

SyntaxDescription
@'...'@Literal here-string — no variable or expression expansion
@"..."@Expanding here-string — $var and $(expr) are substituted
@'...'@ | cmdPipe a literal here-string to any command reading from stdin

Process Substitution and Here-Document Constraints

  • <(), >(), and <<< require Bash. If a script starts with #!/bin/sh on a system where /bin/sh is dash, process substitution and here-strings fail with a syntax error.
  • <<EOF expands variables and command substitutions, while <<'EOF' passes the body literally. Pick the delimiter style deliberately before embedding SQL, templates, or config text.
  • PowerShell here-string closing delimiters must start at column 0. Any leading whitespace turns the closing line into a parser error instead of a terminator.

Process Substitution and Here-Document Recommendations

Bash/Linux | process substitution recommendations | here-docs and tee fan-out

Compare generated outputs with diff <()

Use process substitution when a comparison tool expects filenames but both inputs are generated on the fly. The pattern stays readable and avoids temporary cleanup.

diff <(printf '%s\n' alpha beta) <(printf '%s\n' alpha gamma)
2c2
< beta
---
> gamma

Keep SQL or templates literal with <<'EOF'

Quote the delimiter when the body contains dollar signs, backticks, or placeholder syntax that the shell must not expand. This is the safer default for inline SQL and templates.

cat <<'EOF'
SELECT '$USER' AS literal_user;
EOF
SELECT '$USER' AS literal_user;

Generate config text with expanding here-docs

Use an unquoted delimiter when the block is supposed to interpolate shell variables or command substitutions before it is written. Read the file back immediately if you need to verify the generated values.

tmpdir=$(mktemp -d)
DB_HOST=db.internal
DB_PORT=5432
cat << EOF > "$tmpdir/config.env"
DB_HOST=${DB_HOST}
DB_PORT=${DB_PORT}
PIPELINE_NAME=pipeline_daily
GENERATED_BY=$(printf 'bash-demo')
EOF
cat "$tmpdir/config.env"
rm -rf "$tmpdir"
DB_HOST=db.internal
DB_PORT=5432
PIPELINE_NAME=pipeline_daily
GENERATED_BY=bash-demo

Fan out one stream with tee >() >()

Use output substitution when one producer should feed multiple consumers in a single pass. The verification reads both generated artifacts so the split is explicit.

tmpdir=$(mktemp -d)
printf '%s\n' alpha beta gamma | tee >(gzip > "$tmpdir/data.gz") >(wc -l > "$tmpdir/count.txt") > /dev/null
printf 'count=%s\n' "$(cat "$tmpdir/count.txt")"
gzip -cd "$tmpdir/data.gz"
rm -rf "$tmpdir"
count=3
alpha
beta
gamma

Feed one short stdin value with <<<

Use a here-string for one-off stdin payloads that would otherwise need a trivial pipe. It keeps parsing examples compact and avoids a separate producer command.

read symbol price <<< 'AAPL 145.32'
printf '%s=%s\n' "$symbol" "$price"
AAPL=145.32

PowerShell | process substitution recommendations | temp-file workarounds

Write transient content to a temp file when a command needs a path

PowerShell does not expose process-substitution paths, so a temp file is the direct replacement when a downstream command insists on a filesystem path. Keep creation, use, and cleanup in the same scope.

$tmp = [System.IO.Path]::GetTempFileName()
'alpha','beta' | Set-Content $tmp
Get-Content $tmp
Remove-Item -LiteralPath $tmp -Force
alpha
beta

Process Substitution and Here-Document Troubleshooting

Bash/Linux | process substitution troubleshooting | shell and delimiter errors

Syntax error near unexpected token (

That message usually means the script is running under sh or dash instead of Bash. Process substitution is Bash syntax, so the fix is to rerun the command under Bash or change the shebang to #!/usr/bin/env bash.

sh -c 'diff <(printf ready) <(printf ready)'
sh: 1: Syntax error: "(" unexpected

Variables expanded inside a here document unexpectedly

If a here-document body expands when it should have stayed literal, the delimiter is unquoted. Quote the delimiter to stop the outer shell from touching the body before it reaches the consumer.

name=prod
cat <<EOF
$name
EOF
prod

Variables stayed literal inside a here document

If a here-document body stays literal when it should have expanded, the delimiter is quoted. Switch back to an unquoted delimiter when the body is meant to interpolate shell variables or command substitutions.

name=prod
cat <<'EOF'
$name
EOF
$name

diff <(cmd1) <(cmd2) produced no output

diff is silent when both generated inputs are identical. Check the exit status before assuming the command failed; 0 means the two streams matched.

diff <(printf '%s\n' ready) <(printf '%s\n' ready)
status=$?
echo "exit=$status"
exit=0

PowerShell | here-string troubleshooting | terminator indentation

White space is not allowed before the string terminator

That parser error means the closing '@ or "@ of a here-string is indented. Move the terminator back to column 0 so PowerShell can recognize it as the end of the string.

$bad = @"
$text = @'
hello
  '@
"@
try { Invoke-Expression $bad } catch { $_.Exception.Message }
At line:3 char:3
+   '@
+   ~~
White space is not allowed before the string terminator.

Process Substitution and Here-Document Cross-References

Process Substitution and Here-Document References