Domain: Data and Files

Data and Files

Shell commands for navigating directories, reading and transforming file content, pattern matching, and transferring data across systems.

mindmap
  ((Data and Files))
    (navigation)
    (reading files)
    (grep, regex)
    (awk)
    (sed)
    (date, time)
    (finding files)
    (file manipulation)
    (compression)
    (data transfer)

01-navigation-and-listing

• Directory listing and disk usage
• du vs df discrepancy
• PowerShell equivalents

01-reading-file-contents

• Reading and inspecting files
• Following logs in real-time
• Large log file analysis
• PowerShell equivalents

02-grep-and-pattern-matching

• Basic Pattern Matching
• Recursive search
• Regular Expression Patterns
• Data engineering regex patterns

04-awk-data-processing

• Record and Field Model
• Field Extraction and Formatting
• Filtering and Conditions
• Data Transformation
• Data Engineering Scenarios

03-sed-stream-editing

• Basic Substitution
• In-Place Editing
• Line Selection and Addressing
• Advanced Substitution with Regex

05-date-and-time-handling

• ISO 8601 formats
• Format selection by context
• Bash
• PowerShell
• SQL Server
• Python

03-finding-files

• Find by modification time
• Find by file size
• Find and execute on results
• find vs fd vs locate comparison
• PowerShell equivalents

02-file-manipulation

• Copying, moving, deleting
• Safe delete pattern
• File permissions
• File ownership
• PowerShell equivalents

04-compression

• gzip
• zstd
• tar archives
• Strategy matrix
• PowerShell equivalents

05-data-transfer

• rsync
• Trailing slash gotcha
• scp
• GCE file transfer
• Cloud Storage transfers