Domain: Data Engineering

Data Engineering

File I/O, serialization, APIs, database access, pipelines, testing, and performance — the applied data engineering side of Python and C#.

mindmap
  ((Data Engineering))
    (file IO)
    (serialization formats)
    (web APIs)
    (database access)
    (advanced pipelines)
    (testing)
    (performance)
    (functional pipeline)

File I/O and Serialization

• Read, write, and append files — py · cs
• CSV files — py · cs
• JSON — py · cs
• YAML — py · cs
• Serialization, deserialization, and streams — py · cs
• Encoding and decoding — py · cs

Serialization Formats

• Parquet files — py · cs
• Enterprise message serialization (Protobuf, Avro) — py · cs
• Format performance benchmark — py · cs

Web APIs

• HTTP clients and REST API calls — py · cs
• REST API patterns for data engineering — py · cs
• Building a REST API — py · cs
• Data validation for production APIs — py · cs

Database Access

• SQLite embedded database — py · cs
• SQL Server connectivity — py · cs
• ORM and micro-ORM — py · cs
• DuckDB embedded analytical database — py · cs
• Querying files directly — py · cs

Advanced Pipelines

• Async generators and TPL Dataflow — py · cs
• Parallel API ingestion — py · cs
• Cross-process execution — py · cs

Testing

• Testing philosophy and the testing pyramid — py · cs
• Unit testing (pytest vs xUnit) — py · cs
• Mocking and patching — py · cs
• Test patterns for data engineering — py · cs
• Integration testing with real database — py · cs

Performance and Code Quality

• Timing and benchmarking — py · cs
• Memory profiling and measurement — py · cs
• Big-O complexity and algorithmic thinking — py · cs
• Code smells and anti-patterns — py · cs
• Golden rules of performance — py · cs

Functional Pipeline (End-to-End)

• Pydantic/FluentValidation DTOs — py · cs
• SQL Server schema and medallion tables — py · cs
• Bronze layer ingestion — py · cs
• Silver layer cleaning and enrichment — py · cs
• Gold layer aggregations — py · cs
• Lineage review and audit — py · cs