Five Pillars of Data Engineering
Quote
“The undercurrents of data engineering — security, data management, DataOps, data architecture, orchestration, and software engineering — are the foundation everything else rests on.”
— Joe Reis & Matt Housley, Fundamentals of Data Engineering (2022)
“Reliability is the most important feature. If the system does not do what users need, everything else is irrelevant.”
— Michael Nygard, Release It! (2007)
Summary
This note defines the five load-bearing concerns of production data systems, showing how reliability, observability, efficiency, security, and operability shape architecture decisions long before a team debates tools or implementation details.
Reliability and observability
- Explains why correct repeatable pipelines and visible runtime behavior are the first two pillars, using concrete failure scenarios to show how hidden errors and silent duplication damage trust.
- Treats idempotency, quality gates, structured logs, metrics, and traces as the baseline practices that keep systems explainable under failure.
Efficiency and security
- Covers cost-aware engineering and least-privilege security as architectural concerns rather than afterthoughts, tying bad query design and weak credential handling directly to operational risk.
- Positions money, latency, encryption, identity, and secret handling as part of the same senior-engineering decision surface.
Operability and vault mapping
- Defines operability as the ability for another engineer to run, debug, and change the system safely, then maps each pillar to deeper notes elsewhere in the vault.
- Uses runbooks, naming, deployment discipline, and documentation as system qualities rather than documentation chores.
Operations and safety
- Warnings: each pillar section includes concrete failure modes that show how one neglected concern can invalidate otherwise good technical work.
- Recommendations: treat the five pillars as a recurring architecture review checklist before and after a system goes live.
Glossary
Engineering pillar
A recurring system quality that should shape design choices across the entire platform rather than only one component.
It matters here because the note frames senior data engineering as balancing a small set of durable concerns instead of chasing isolated tool skills.
Design lens, not feature
A pillar is useful because it cuts across languages, clouds, and products. It changes how you evaluate every architecture choice.
Reliability
The property that a pipeline produces the intended result consistently and can be rerun safely when something fails.
It matters here because wrong data is often more damaging than missing data, especially when failures stay silent.
Silent bad data
A pipeline that appears healthy while duplicating or corrupting records is worse than one that fails loudly and stops downstream publication.
Observability
The ability to infer what a system is doing from its emitted metrics, logs, traces, and supporting metadata.
It matters here because production problems become solvable only when operators can see stage, state, and failure context quickly.
Debugging surface
Observability is what closes the gap between a generic error and a short, repeatable investigation path.
Efficiency
The practice of minimizing wasted compute, storage, and human effort while still meeting functional goals.
It matters here because cloud bills, slow queries, and overbuilt pipelines are architectural failures, not only optimization tasks.
Cost is architecture
Efficiency decisions compound over time. A small waste inside a scheduled pipeline becomes a permanent operating cost.
Security
The set of controls that protect data, identities, secrets, and system boundaries from unauthorized access or misuse.
It matters here because data platforms routinely handle regulated, proprietary, or business-critical information.
Baseline not hardening
Credential discipline, encrypted transport, and scoped identities are not optional upgrades. They are the minimum safe operating standard.
Operability
The quality that makes a system understandable, supportable, and recoverable by someone other than its original author.
It matters here because real systems are maintained under pressure, often by on-call engineers who did not design them.
Future operator matters
If a system requires tribal knowledge to restart or diagnose, its architecture is incomplete even if the code is technically sound.
Trade-off
A decision where improving one quality or constraint usually costs something in another area such as latency, simplicity, or price.
It matters here because the five pillars are not independent; stronger security, lower cost, and richer observability all interact.
No free architecture
Senior engineering is largely the discipline of making trade-offs explicit instead of pretending one choice optimizes everything.
Failure mode
A concrete way a system can break, misbehave, or produce misleading outcomes under real operating conditions.
It matters here because the note teaches the pillars through failure cases rather than through abstract slogans alone.
Design from breakage
Thinking in failure modes forces architecture decisions to account for what happens when reality departs from the happy path.
Reliability
Your pipeline must produce correct data, every time. This means idempotent transforms, atomic loads, proper error handling, and retry logic that does not silently corrupt state. A pipeline that works 99% of the time is a pipeline that lies to its consumers 3.65 days per year.
Without Reliability
A pipeline runs on Tuesday but silently inserts duplicate rows because the MERGE key was wrong. Nobody notices — the row count looks normal. On Friday, a portfolio manager spots that BASF appears twice in the index, inflating its weight. The published index was wrong for three days. Three days of wrong NAV calculations for every ETF tracking it. Idempotent transforms, quality gates, and lineage tracking prevent this.
Build reliability in from the design stage — idempotency, quality gates, lineage
Use MERGE upserts with a correct business key (not an IDENTITY column) to make every load idempotent. Add a post-stage quality gate that checks for duplicate
(symbol, date)pairs before the run completes. Trackbatch_idand row counts in a lineage table so any anomaly is traceable to its exact pipeline run within seconds.
Observability
You cannot fix what you cannot see. Every system you operate needs metrics (how much), logs (what happened), and traces (where did time go). The gap between “it works on my machine” and “it works in production” is entirely filled by observability. See datadog-architecture-overview and cloud-logging.
Without Observability
The pipeline fails at 2 AM. The on-call engineer sees “process exited with code 1” in the Airflow log. No structured logging, no trace ID, no metric showing which stage failed or what the input looked like. They SSH into the VM, grep through 50 MB of unstructured logs, find the error 45 minutes later. With structured logging, the Datadog dashboard shows the failing stage, input row count, and error message within 30 seconds.
Emit structured logs with stage, batch_id, row counts, and error context at every step
Use a structured logging pattern:
log.error({"stage": "silver_enrich", "batch_id": batch_id, "input_rows": n, "error": str(e)}). Ship logs to Datadog or Cloud Logging with a dashboard that surfaces failing stage and error type instantly. The on-call engineer should never need to SSH into a VM to diagnose a pipeline failure.
Efficiency
Cloud resources cost real money. A query that scans 10 TB when it could scan 10 GB is not just slow — it is a $50 billing event that happens every time someone runs it. Senior engineers think in dollars-per-query, IOPS-per-transaction, and cold-start-latency-per-invocation. See querying-and-cost-optimization and cost reference.
Without Efficiency
A BigQuery query scans a 2 TB table to count rows for one date. It costs 29,000 for a query that could cost $0.01 with partition pruning. The fix is one line:
WHERE _PARTITIONDATE = CURRENT_DATE(). But nobody looked at the billing dashboard until the invoice arrived.
Set up a BigQuery billing export and review it weekly
Enable BigQuery billing export to a BigQuery dataset. Create a dashboard showing cost-per-query by user and by job. Set a Cloud Billing budget alert at 80% of the monthly threshold. Review the top-10 most expensive queries each week and add partition filters or clustering where missing. See querying-and-cost-optimization for the optimization techniques.
Security
The data you move often contains financial information, personal identifiers, or proprietary signals. Least-privilege access, encrypted connections, parameterized queries, and credential rotation are not optional hardening steps — they are baseline professional standards. See service-accounts-and-iam and tde-encryption.
Without Security
A developer downloads a service account key for “quick local testing” and commits it to a public GitHub repo. The key has BigQuery Admin permissions. A scanner finds it within 4 hours. The attacker exports the entire dataset — financial positions, ESG scores, client weights. Workload Identity Federation eliminates key files entirely. The key that doesn’t exist can’t be leaked.
Use Workload Identity Federation — eliminate all service account key files
Configure GitHub Actions and Cloud Run to authenticate via WIF instead of key files. A federated identity has no downloadable credential — there is nothing to commit, nothing to leak, nothing to rotate. See secrets-management > GitHub Actions — Workload Identity Federation (Keyless) for the setup pattern.
Operability
Every system you build will eventually be operated by someone who is not you, possibly at 3 AM during an outage. Clear naming conventions, documented runbooks, structured logging, and predictable deployment processes are what make a system operable. If your successor needs to read your mind to operate your system, you have failed as an engineer.
Without Operability
The engineer who built the pipeline leaves. The replacement opens the DAG file and sees 47 tasks with names like
task_7b,step_final_v2, andDO_NOT_CHANGE. No runbook, no architecture doc, no structured naming. It takes two weeks to understand what the pipeline does and three months to feel confident making changes. Clear naming, documented runbooks, and structured logging make this a two-day onboarding.
Write the runbook before the pipeline goes live — not after
For every DAG, write a one-page runbook covering: what it does, what it depends on, what a failure looks like, and how to restart it safely. Use descriptive task names (
extract_daily_ohlcv, nottask_3). Document oncall steps in the same repo as the code so they stay in sync. See airflow-dag-patterns for naming conventions and defensive-scripting for structured error output patterns.
How These Pillars Map to the Vault
Reading This Vault
This is a reference, not a novel. Jump to whatever section matches the problem in front of you. Each note is self-contained. Wikilinks point you to related material when concepts overlap.