04 - Compression

10 min read

Compression

Quote

“There is no compression algorithm for experience.”

  • Andy Jassy, AWS re:Invent keynote (2012)

Compression decisions split into two questions: are you compressing one file or a whole directory tree, and does compatibility matter more than throughput? The examples below keep those choices separate and show live output for each tool family.

Linux compression workflows

Linux | gzip and pigz | single-file gzip workflows

gzip is the compatibility baseline for Linux and Unix-like environments. pigz keeps the same .gz format but parallelizes compression work across CPU cores.

Compress a stream with gzip

gzip -c writes compressed bytes to stdout instead of replacing the source file. Piping the result into wc -c gives a fast way to compare output size without creating a file on disk.

Run the commands in this section to compress a stream with gzip.

gzip -c /etc/services | wc -c
5379

Inspect a .gz file without extracting it

gzip -l reports compressed size, uncompressed size, and ratio for an existing gzip member. Use it when you need to estimate payload size or confirm that a file is actually gzip before decompressing it.

Run the commands in this section to inspect a .gz file without extracting it.

gzip -l /usr/share/man/man1/printf.1.gz
         compressed        uncompressed  ratio uncompressed_name
               1273                2335  46.3% /usr/share/man/man1/printf.1

Keep the gzip format but parallelize compression with pigz

pigz -c emits the same gzip format as gzip -c. On the same input used above, the byte count is identical, which is why pigz works as a drop-in replacement for .gz workflows.

Run the commands in this section to keep the gzip format but parallelize compression with pigz.

pigz -c /etc/services | wc -c
5379

Linux | zstd and lz4 | alternative single-file codecs

When downstream systems do not require .gz, compare a modern codec against gzip on representative data. Small files often behave differently from large CSV, JSON, or log batches because container overhead matters more.

Compress the same input with zstd

This command sends /etc/services through zstd at its default level and counts the compressed bytes. The result is close to gzip on this small input, which is exactly why ratio claims should be validated against real payloads.

Run the commands in this section to compress the same input with zstd.

zstd -cq /etc/services | wc -c
5559

Compress the same input with lz4

lz4 trades ratio for speed. The larger byte count here is expected and is usually acceptable only when decompression latency matters more than storage or network cost.

Run the commands in this section to compress the same input with lz4.

lz4 -cq /etc/services | wc -c
8056

Linux | tar | archive directory trees before compression

A directory tree has to be archived before it can be compressed as one unit. tar handles the packaging step; the codec flag determines how that tar stream is compressed.

Create a gzip-compressed tar stream

This example archives /etc/hosts and /etc/services into one gzip-compressed tar stream and measures the resulting byte count. The key point is that tar is operating on multiple paths, not on a single file.

Run the commands in this section to create a gzip-compressed tar stream.

tar -C /etc -czf - hosts services | wc -c
5752

Create a zstd-compressed tar stream

tar --zstd swaps the compression algorithm while keeping the same archive structure. On this tiny two-file archive, the zstd-wrapped tar stream is slightly larger, which is a reminder to benchmark instead of assuming.

Run the commands in this section to create a zstd-compressed tar stream.

tar -C /etc --zstd -cf - hosts services | wc -c
5982

Inspect archive paths before extraction

List an unfamiliar tarball with tar -tf before you extract it, and prefer extracting into a fresh directory with -C. That quick check surfaces unexpected top-level paths before the archive writes into your current tree.

PowerShell compression workflows

PowerShell | Compress-Archive | built-in ZIP output

Compress-Archive is the built-in choice for ZIP files on Windows. It is convenient for handoffs and ad hoc packaging, but it is not a general compression front end: it writes ZIP only, skips hidden items, and inherits the ZipArchive 2 GB per-file limit.

ZIP input shape changes the result

Compress-Archive has three quiet constraints that matter in automation:

  • Hidden files and folders are ignored.
  • The underlying ZipArchive API limits each archived file to 2 GB.
  • If you feed it a recursive Get-ChildItem result, both directories and files can be added, which duplicates entries. Archive the root path directly when you want one clean ZIP tree.

Create a ZIP archive with Compress-Archive

-PassThru makes the cmdlet emit the created archive object, which gives you immediate verification without a second command. -CompressionLevel Fastest is usually the right starting point for already structured data that will be moved again soon.

Run the commands in this section to create a ZIP archive with Compress-Archive.

Compress-Archive -Path "$PSHOME\pwsh.exe" -DestinationPath "$env:TEMP\vault-compression-demo.zip" -CompressionLevel Fastest -Force -PassThru | ForEach-Object { "{0}`t{1}" -f $_.Name, $_.Length }
vault-compression-demo.zip	132805

PowerShell | 7-Zip | explicit gzip workflows

When the workflow needs .gz rather than .zip, use the explicit 7-Zip binary path in automation. That avoids PATH ambiguity and makes it clear which tool is responsible for the archive format.

Create a gzip file with the explicit 7z.exe path

This command writes a gzip member from pwsh.exe and filters the tool output down to the confirmation lines that matter in automation logs.

Run the commands in this section to create a gzip file with the explicit 7z.exe path.

& 'C:\Program Files\7-Zip\7z.exe' a -tgzip "$env:TEMP\vault-compression-demo.gz" "$PSHOME\pwsh.exe" | Select-String -Pattern 'Archive size:','Everything is Ok' | ForEach-Object { $_.Line.Trim() }
Archive size: 104191 bytes (102 KiB)
Everything is Ok

List the member stored in that gzip file

7z l is the inspection step before extraction. Here it confirms that the gzip member contains pwsh.exe and reports both logical and compressed size.

Run the commands in this section to list the member stored in that gzip file.

& 'C:\Program Files\7-Zip\7z.exe' l "$env:TEMP\vault-compression-demo.gz" | Select-String -Pattern 'Type = gzip','Name$','pwsh\.exe$' | ForEach-Object { $_.Line }
Type = gzip
   Date      Time    Attr         Size   Compressed  Name
2026-03-12 03:11:00 .....       295456       104191  pwsh.exe

PowerShell | GZipStream | scripted gzip

GZipStream is the right abstraction when compression needs to stay inside a PowerShell script. The important implementation detail is to connect input and output streams directly instead of materializing the whole file as one byte array.

Compress a file with stream-based GZipStream

This example opens the source file as a stream, copies it into a gzip stream, and emits the created .gz file information. The command stays in-process and avoids the ReadAllBytes() pattern that scales poorly on large files.

Run the commands in this section to compress a file with stream-based GZipStream.

$source = Join-Path $PSHOME 'pwsh.exe'; $target = Join-Path $env:TEMP 'vault-compression-gzipstream.gz'; $input = [System.IO.File]::OpenRead($source); $output = [System.IO.File]::Create($target); $gzip = [System.IO.Compression.GZipStream]::new($output, [System.IO.Compression.CompressionLevel]::Fastest); $input.CopyTo($gzip); $gzip.Dispose(); $output.Dispose(); $input.Dispose(); Get-Item $target | ForEach-Object { "{0}`t{1}" -f $_.Name, $_.Length }
vault-compression-gzipstream.gz	132709

Selecting the format

  • Use gzip or pigz when the downstream contract is explicitly .gz, especially for logs and interchange files that must decompress everywhere.
  • Use zstd for modern intermediate files and archives when both ends support it, then benchmark level and ratio on representative data before locking in a default.
  • Use lz4 when read latency or CPU budget dominates storage cost, such as hot local caches or very short-lived transport buffers.
  • Use tar whenever the input is a directory tree; compression comes after archiving, not instead of it.
  • Use Compress-Archive for built-in ZIP workflows on Windows, 7-Zip for explicit gzip creation and inspection, and GZipStream when the compression step must stay inside script logic.

The examples above also show why blanket rules are risky. On the captured /etc/services and /etc/{hosts,services} samples, gzip produced slightly smaller outputs than zstd because the inputs were small enough for container overhead to dominate. Measure the data you actually ship.

Cross-references

  • file-manipulation - moving, renaming, and cleaning up generated archives
  • finding-files - locating old archives and bulk-compressing matched files
  • data-transfer - combining compression with remote copies and streaming transfers

Graph View

Backlinks