“Good data architecture serves business requirements with a common, widely reusable set of building blocks while maintaining flexibility and making appropriate trade-offs.”
— Joe Reis & Matt Housley, Fundamentals of Data Engineering (2022)
Summary
This note defines the data lake as the low-cost, schema-on-read storage layer for multi-format data, then shows how zone design, file layout, governance, and platform choices determine whether that flexibility becomes a durable asset or an unusable swamp.
Schema model and zone design
Contrasts schema-on-write and schema-on-read, then uses landing, cleansed, and curated zones to show where raw ingest ends and enforced structure begins.
Treats zone boundaries as the real quality contract, with immutable raw storage, validated conformance, and analytics-ready publication serving distinct access patterns.
Storage layout and file choices
Covers Hive-style partitioning, naming conventions, file size targets, and storage-format selection so lake queries stay prunable, compressible, and operationally predictable.
Connects partition strategy, small-file control, and columnar formats directly to both performance and long-term storage cost.
Governance, anti-patterns, and cloud implementations
Explains cataloging, lineage, access control, and PII handling, then uses the data swamp anti-patterns to show what breaks when a lake is only cheap storage with no discipline.
Compares cloud lake implementations and walks through a GCP pattern built from GCS, BigQuery external tables, and Dataproc for heavier transformation work.
Operations and safety
Warnings: schema-on-read is deferred enforcement rather than no enforcement, and weak partitioning, uncontrolled small files, or missing ownership quickly turn the lake into a swamp.
Recommendations: enforce schemas at zone transitions, catalog every dataset early, partition mostly by time, and compact incremental files before query cost and latency drift upward.
Glossary
Data lake
A centralized storage layer that keeps structured, semi-structured, and unstructured data in low-cost object storage, usually in its original or lightly standardized formats.
It matters here because the entire note explains how to make that raw flexibility operationally safe instead of letting it decay into unmanaged storage.
Cheap does not mean simple
The storage bill may be low, but the engineering discipline needed to keep the lake queryable and trustworthy is high.
Schema-on-read
A data access model where structure is interpreted at query time instead of being fully enforced before data lands in storage.
It matters here because it is the defining flexibility of the lake and the reason zone transitions must carry the real validation burden.
Deferred enforcement
Schema-on-read does not remove schema work. It postpones it, which means errors can accumulate silently until a consumer query fails or returns bad results.
Landing / raw zone
The immutable append-only area that stores exact source-system payloads as they arrived, without correction or normalization.
It matters here because the raw zone is the audit trail and reprocessing source for every later lake transformation.
Preserve the original record
If upstream sends malformed data, keep it in landing anyway. The safe pattern is to quarantine or reject later, not to erase the original arrival.
Cleansed / conforming zone
The lake layer where data is validated, deduplicated, standardized, and rewritten into formats with declared schemas.
It matters here because this is where the lake stops being passive storage and starts enforcing reusable contracts.
Do not leave JSON here
A cleansed zone that still holds arbitrary CSV or JSON has not really enforced a contract. It only moved the raw files to a new folder.
Curated / analytics zone
The business-facing layer that publishes optimized, governed datasets for analysts, BI, or downstream serving systems.
It matters here because curated data is the point where lake storage becomes directly useful to non-engineering consumers.
Fit for access patterns
Curated data is not only cleaner. It is also shaped for dominant query patterns, discoverability, and governed access.
Hive-style partitioning
A directory layout convention that encodes partition keys in paths such as year=2026/month=03/day=22/ so engines can discover and prune partitions automatically.
It matters here because partition layout is a primary performance lever for lake reads across Spark, BigQuery, Athena, and similar engines.
Cardinality trade-off
Very high-cardinality partitions create too many directories, while very low-cardinality partitions barely prune anything. Time-based partitioning is the safest default.
Small files problem
The performance and metadata overhead created when incremental writes produce huge numbers of tiny objects instead of fewer well-sized columnar files.
It matters here because lake performance and cost can degrade even when the data volume itself is reasonable.
Metadata becomes the bottleneck
Query engines can spend more time listing and opening files than reading actual data when partitions accumulate thousands of tiny objects.
Data swamp
A failed lake implementation where datasets exist in storage but lack clear schema, ownership, cataloging, retention, or trust.
It matters here because the note treats swamp avoidance as the real architectural challenge of operating a lake.
Cheap storage trap
Teams often mistake successful ingestion for successful architecture. If no one can discover, validate, or safely reuse the data, the lake is already failing.
External table
A query-engine table definition that references files stored outside the engine’s native managed storage.
It matters here because the note uses BigQuery external tables as the bridge between lake files in GCS and SQL-based analytical access.
Compute without copying
External tables let analysts work through a familiar SQL surface while the physical data remains in object storage.
Data catalog
A metadata system that records datasets, schemas, owners, lineage, and discovery attributes for the lake.
It matters here because governance starts collapsing as soon as files exist without searchable metadata and ownership context.
Register early
Retrofitting a catalog after dozens of producers have already written unmanaged files is much harder than requiring registration at dataset creation time.
Schema-on-Write vs Schema-on-Read
Understanding this distinction is the architectural foundation of the data lake concept.
Dimension
Schema-on-Write (Data Warehouse)
Schema-on-Read (Data Lake)
When schema is defined
Before loading (DDL must exist first)
At query time (schema inferred or declared)
Data transformation
Required before loading
Optional — raw data stored as-is
Flexibility
Low — schema changes require ALTER TABLE
High — add new data without schema changes
Query performance
High — optimized columnar storage, statistics
Variable — depends on file format and partitioning
Data quality enforcement
At load time (rejects bad data)
At query time (bad data accepted, may cause errors)
Storage cost
Higher (specialized storage, compute attached)
Lower (commodity object storage)
Best for
Known query patterns, BI dashboards
Exploration, ML training data, multi-format sources
Schema-on-Read Is Not Free
Schema-on-read means the lake accepts anything — but it also means bad data silently enters the lake and corrupts downstream queries. A well-run data lake enforces quality at zone boundaries (see the zone architecture below), not at every raw file. The real discipline is enforcing schemas at the transition from raw to curated zones, not at ingest.
Zone Boundary Enforcement
Accept any format in the landing zone (raw, immutable copy) but enforce a declared schema at the landing → cleansed transition. Use Spark’s DROPMALFORMED mode or a Python Pydantic validator to reject or quarantine records that fail the schema. Any quarantined record lands in a _rejected/ partition alongside the cleansed data, preserving the audit trail without polluting the cleansed zone.
Zone Architecture
The canonical data lake organizes storage into zones (also called layers or tiers), each with a defined quality level, access pattern, and governance contract. The zone concept maps directly to the medallion-architecture (Bronze = Landing/Raw, Silver = Cleansed, Gold = Curated).
flowchart TD
src["Source Systems"]
landing["LANDING / RAW ZONE (Bronze)<br/>Exact copy, no transformation<br/>Immutable — write once<br/>All formats: JSON, CSV, Parquet, XML<br/>Retained 90 days to permanent<br/>Access: pipeline service accounts only"]
cleansed["CLEANSED / CONFORMING ZONE (Silver)<br/>Validated, deduplicated, schema-enforced<br/>Parquet / Avro only<br/>Hive-style date partitioning<br/>PII masked or tokenized<br/>Access: data engineers + approved tooling"]
curated["CURATED / ANALYTICS ZONE (Gold)<br/>Business-ready, query-optimized<br/>Columnar, compressed, partitioned<br/>Denormalized for common patterns<br/>Access: analysts, BI, data science<br/>Exposed via external tables or DWH"]
src --> landing
landing -->|"validation +<br/>cleaning pipeline"| cleansed
cleansed -->|"aggregation +<br/>enrichment pipeline"| curated
style src fill:#1a1a2e,stroke:#7aa2f7,color:#fff
style landing fill:#1a1a2e,stroke:#e0af68,color:#fff
style cleansed fill:#1a1a2e,stroke:#bb9af7,color:#fff
style curated fill:#1a1a2e,stroke:#9ece6a,color:#fff
Landing / Raw Zone
The landing zone is an append-only, immutable record of everything that arrived from source systems. Never delete from this zone and never modify files once written. If a source sends bad data, that bad data must be preserved in landing — it is the audit trail.
What goes here
API responses as raw JSON files
Database exports as CSV or bulk format
Event streams from Pub/Sub written as Avro or JSON
Third-party data feeds in whatever format the vendor sends
Retention: Minimum 90 days for re-processing; many organizations retain permanently for auditability.
Access: Restrict to pipeline service accounts. Analysts should never query the landing zone directly — raw data is unvalidated and may change format without notice.
In a GCP lake, mount the curated zone as BigQuery external tables. Analysts get the familiar BigQuery SQL interface and cost controls (partition pruning, dry runs) while the data physically lives in GCS. When query performance demands it, materialize the most-queried external tables into native BigQuery tables. See querying-and-cost-optimization and dataset-and-table-management for setup.
File Organization and Naming Conventions
Consistent file organization is not aesthetic — it determines whether partition discovery works, whether object listing is fast, and whether data consumers can predict where files live.
Hive-Style Partitioning
The de facto standard for data lake file organization. Partition paths use key=value pairs that query engines (Spark, Presto, BigQuery, Athena) automatically discover and use for partition pruning.
Query engines read partition columns from path, not from file
# PySpark / Dataproc: reads partition key=value from path automaticallydf = spark.read.parquet("gs://example-data-lake-cleansed/equity-prices/")df.filter(df.year == 2026).filter(df.month == 3)# Only reads year=2026/month=03/ directories — no full scan
Partition Cardinality Balance
Partition on columns with moderate cardinality. Date (YYYY/MM/DD) is the most common and usually ideal. Avoid partitioning by high-cardinality columns (instrument_id with 50,000 values creates 50,000 directories — object listing becomes the bottleneck). Avoid partitioning by low-cardinality columns (market = [US, EU, APAC] → only 3 directories, no pruning benefit for most queries).
Recommended Partition Strategy
Partition first by date (year/month/day for daily data, year/month for monthly aggregates). If a secondary dimension is needed, choose one with 10–500 distinct values (e.g., asset class, region, exchange). For very high-cardinality secondary dimensions like instrument_id, use clustering (BigQuery) or sorting within Parquet row groups instead of directory-level partitioning.
File Naming Conventions
# Pattern: {entity}_{date}_{sequence}.{format}[.{compression}]
equity_prices_20260322_001.parquet
trades_20260322_143000_001.parquet.gz
ohlcv_20260322_001.snappy.parquet
# Avoid:
data.parquet # not unique, cannot be identified
equity prices (1).parquet # spaces, parentheses — breaks many tools
EQUITY_PRICES_2026-03-22.PARQUET # case inconsistency, dashes in date
Rules
All lowercase, no spaces, no special characters except underscores and dots
Include date (YYYYMMDD) or datetime (YYYYMMDDHHMMSS) in filename
Include sequence number for multi-file batches (_001, _002)
Always include the format extension (.parquet, .avro, .json.gz)
Include compression codec in extension if non-obvious (.snappy.parquet, .gz)
File Size Optimization
Object storage (GCS, S3) has throughput proportional to object size. Many small files degrade performance dramatically — this is the “small files problem.”
Landing zones that receive incremental files will accumulate thousands of small files over time. Run a weekly compaction job that merges small Parquet files by partition into optimal-size consolidated files. open-table-formats (Apache Iceberg, Delta Lake) manage this automatically through their OPTIMIZE / REWRITE DATA FILES operations.
Storage Formats: When to Use Each
See serialization-formats for detailed encoding mechanics and compression codec comparison. This section covers the decision criteria specific to data lake storage.
Format
Schema embedded
Columnar
Splittable
Best use case in a lake
CSV
No
No
Yes (line-based)
Landing zone raw ingest only; human-readable
JSON
No
No
Yes (line-delimited)
Landing zone API responses; debug/inspection
JSON Lines (NDJSON)
No
No
Yes
Streaming event landing; Pub/Sub output
Parquet
Yes
Yes
Yes (row groups)
Cleansed + curated zones; analytical queries
Avro
Yes
No
Yes
Streaming / Kafka; schema evolution focus
ORC
Yes
Yes
Yes
Hive-ecosystem legacy; Presto/Trino on AWS
Decision rules
Landing zone: Whatever format the source sends (preserve exactly)
Cleansed zone: Parquet (analytics) or Avro (if streaming/Kafka origin)
Curated zone: Parquet (compressed, partitioned, clustered)
Never Use CSV or JSON in the Curated Zone
CSV and JSON have no embedded schema, no columnar storage, and no compression interoperability. A 10 GB CSV file in the curated zone will be read end-to-end for every query. The same data as Parquet with Snappy compression is typically 2–5 GB and scanned 3–10x faster because query engines read only the relevant columns. See serialization-formats for the full format comparison.
Convert at the Cleansed Zone Boundary
Run a conversion step as the final action of every cleansing pipeline: read the raw CSV/JSON from landing, validate the schema, and write the output as Parquet with Snappy compression and Hive-style date partitioning. The cleansed zone should contain only Parquet (or Avro for Kafka-origin data). Any analyst or BI tool that claims to need CSV can be served by a one-off export, not by storing CSV in the lake permanently.
Parquet configuration for data lake
import pyarrow as paimport pyarrow.parquet as pq# Write Parquet with optimal settings for analyticstable = pa.Table.from_pandas(df)pq.write_to_dataset( table, root_path="gs://example-data-lake-cleansed/equity-prices/", partition_cols=["year", "month", "day"], compression="snappy", # fast decompression; use zstd for higher compression ratio row_group_size=128 * 1024, # 128K rows per row group (balance of read speed vs overhead) write_statistics=True, # column statistics enable predicate pushdown use_dictionary=True, # dictionary encoding for low-cardinality string columns filesystem=gcs_filesystem,)
Data Lake Governance
An ungoverned data lake inevitably becomes a data swamp. The three most common paths to a swamp:
Data lands without metadata — no one knows what’s there
No schema enforcement — downstream queries fail without warning
No retention policy — the lake grows without bound, costs spiral
Data Cataloging
A data catalog is the index of the data lake: what tables exist, where they live, what their schemas are, who owns them, when they were last updated, and what they contain.
Minimum viable catalog entry for each dataset
name: equity_prices_dailydescription: "Daily OHLCV prices for equity instruments sourced from the yfinance API"owner: data-engineering@org.comlocation: gs://example-data-lake-cleansed/equity-prices/format: parquetpartition_columns: [year, month, day]schema: - name: instrument_id type: STRING description: "Exchange-specific ticker symbol" - name: trade_date type: DATE description: "Trading date (ISO 8601)" - name: open type: FLOAT64 - name: high type: FLOAT64 - name: low type: FLOAT64 - name: close type: FLOAT64 - name: volume type: INT64classification: internalpii: falserefresh_cadence: dailysla_available_by: "08:00 UTC"lineage_upstream: [yfinance-api]lineage_downstream: [equity_prices_curated, fact_prices_bq]
GCP-native catalog options
Tool
Scope
Best for
BigQuery Data Catalog (Dataplex)
GCP-native, BigQuery + GCS
GCP-only shops, tight BQ integration
Apache Atlas
Multi-cloud, open source
Enterprise, Hadoop ecosystem
Collibra / Alation
Commercial, enterprise
Large organizations with data governance teams
DataHub (LinkedIn)
Open source, multi-cloud
Mid-size engineering teams
Data Lineage
Lineage tracks the provenance of each dataset: what were its inputs, what transformation produced it, what does it feed downstream. Without lineage, a bug in a source system cannot be traced to impacted downstream reports.
Implement Lineage from Day One
Retrofitting lineage onto an existing lake is vastly more expensive than building it in from the start. Even a simple metadata file (lineage.json) alongside each dataset — listing upstream source names and the pipeline job that produced it — provides enormous value when something breaks.
Minimum lineage metadata approach (practical for small teams)
Granting roles/storage.objectViewer at the project level gives access to all buckets in the project. Assign bucket-level IAM bindings to enforce zone separation. Use service-accounts-and-iam’s condition-based IAM for attribute-level access control.
Bucket-Level IAM Pattern
Create one service account per pipeline stage (e.g., sa-landing-writer, sa-cleanse-reader, sa-curate-writer) and bind each to its specific bucket with the minimum required role. Analysts get roles/storage.objectViewer on the curated bucket only. Enforce this via Terraform so IAM bindings are code-reviewed and version-controlled, not applied ad hoc via the console.
PII Handling in the Lake
Personally Identifiable Information (PII) in the lake requires a clear handling strategy.
Options in order of preference
Tokenization — Replace PII with a reversible token. The token → PII mapping lives in a secure vault (GCP Secret Manager). Data in the lake contains only tokens.
Pseudonymization — Hash the PII with a secret salt. Irreversible without the salt. Suitable for analytics where re-identification is not needed.
Masking — Replace PII with a placeholder (***, REDACTED). Irreversible. Suitable for logs and audit trails.
Column-level encryption — Encrypt the PII column with a KMS key. Authorized users decrypt; others see ciphertext.
PII should never appear in the curated zone. Apply the chosen technique at the cleansed zone boundary (the cleansing pipeline) so the curated zone is PII-free by design.
Data Lake Anti-Patterns: The Data Swamp
A data lake becomes a data swamp when it grows without governance. Swamps are characterized by data that cannot be trusted, cannot be found, and cannot be deleted.
Anti-Pattern 1: No Schema Enforcement
Symptom: Data lands in arbitrary formats. Downstream queries fail with type errors or return wrong results.
Fix: Enforce a schema contract at the cleansed zone boundary. Reject or quarantine data that fails validation. Use Parquet’s embedded schema as the enforcement mechanism — write Parquet with a declared schema and any source data that doesn’t conform to the schema errors at write time.
Anti-Pattern 2: No Cataloging
Symptom: “Where is the price data?” requires asking a specific person. New team members cannot discover what exists.
Fix: Register every dataset in a catalog at write time, not as an afterthought. Automate catalog updates as part of the pipeline completion step.
Anti-Pattern 3: No Retention Policy
Symptom: The lake grows indefinitely. Storage costs compound monthly. “We might need it someday” is the only retention policy.
Fix: Define explicit retention periods per zone and per dataset. Implement GCS lifecycle rules to automatically transition and delete data. See gcs-buckets-and-lifecycle for lifecycle rule syntax.
Symptom: All data — landing, cleansed, curated — lives in a single bucket. Access control is all-or-nothing. Zone boundaries are meaningless.
Fix: One bucket per zone. This maps zone boundaries to security boundaries. IAM is applied at the bucket level, and storage class / lifecycle rules are configured independently per zone.
Symptom: Re-running the pipeline for a date creates duplicate files or doubles row counts.
Fix: Design every pipeline write as idempotent. For Parquet in GCS, the most reliable approach is to write to a temporary path and then atomically rename (or use the idempotent-pipeline-design TRUNCATE + RELOAD pattern):
# Idempotent Parquet write: write to temp, rename to finaltemp_path = f"gs://bucket/_temp/{run_id}/equity-prices/"final_path = f"gs://bucket/cleansed/equity-prices/year={year}/month={month}/day={day}/"# Write to tempdf.to_parquet(temp_path)# Delete existing final path contentgcs_client.delete_blobs_with_prefix(final_path)# Move temp → final (using GCS rewrite for cross-prefix move)gcs_client.move_blobs(temp_path, final_path)
Cloud Data Lake Implementations Comparison
Dimension
GCS (Google Cloud Storage)
S3 (AWS)
ADLS Gen2 (Azure)
Storage pricing (standard)
$0.020/GB/month (US multi-region)
$0.023/GB/month (US East)
$0.018/GB/month (LRS)
Storage classes
STANDARD, NEARLINE, COLDLINE, ARCHIVE
Standard, Infrequent Access, Glacier, Glacier Deep
GCS and S3 are functionally equivalent for most data lake workloads. The key practical difference is the query engine integration: GCS plugs natively into BigQuery (zero-copy external tables), while S3 plugs natively into Athena and Redshift Spectrum. Choose based on which warehouse engine you use, not on raw storage characteristics.
Data Lake vs Data Warehouse Comparison
Dimension
Data Lake
Data Warehouse
Data types
All (structured, semi-structured, unstructured)
Structured only
Schema
Schema-on-read
Schema-on-write
Storage cost
Very low (object storage)
Higher (specialized storage)
Query performance
Variable (depends on format + partitioning)
High (always optimized, statistics maintained)
Data quality
Variable — quality enforced at zone boundaries
High — enforced at load time
Concurrency
High (object storage is infinitely scalable)
Limited by compute (slots, DWU, virtual warehouses)
SQL support
Via query engines (Presto, Spark, BQ external tables)
Known query patterns, BI dashboards, governed reporting
Lake + Warehouse = Lakehouse
Modern architectures combine both: a data lake for low-cost raw storage, feeding a data warehouse or lakehouse layer for governed analytical queries. The open-table-formats (Apache Iceberg, Delta Lake) blur this boundary further — open table formats bring warehouse-grade ACID transactions and schema enforcement to object storage, creating the “lakehouse” architecture. The medallion-architecture is a practical implementation pattern that spans both.
GCP Data Lake Implementation: GCS + BigQuery + Dataproc
A complete GCP-native data lake uses GCS for storage, BigQuery for SQL-on-lake queries, and Dataproc (managed Spark) for heavy transformation jobs.
Setting Up the Three-Zone Lake on GCS
# Create three zone buckets with appropriate storage classes and access controls# Landing zone: STANDARD, 90-day auto-delete, restricted accessgcloud storage buckets create gs://example-data-lake-landing \ --location=EU \ --default-storage-class=STANDARD \ --uniform-bucket-level-access# Cleansed zone: STANDARD, 1-year retention, data engineering accessgcloud storage buckets create gs://example-data-lake-cleansed \ --location=EU \ --default-storage-class=STANDARD \ --uniform-bucket-level-access# Curated zone: STANDARD → NEARLINE after 365d, analyst accessgcloud storage buckets create gs://example-data-lake-curated \ --location=EU \ --default-storage-class=STANDARD \ --uniform-bucket-level-access
BigQuery external tables let analysts query GCS Parquet files using standard SQL without loading data into BigQuery. Costs are based on bytes scanned from GCS (priced at BigQuery on-demand rates).
-- Create a BigQuery external table pointing to GCS ParquetCREATE OR REPLACE EXTERNAL TABLE `example-project.lake_curated.equity_prices_ext`WITH PARTITION COLUMNS ( year INT64, month INT64, day INT64)OPTIONS ( format = 'PARQUET', uris = ['gs://example-data-lake-curated/equity-prices/*.parquet'], hive_partition_uri_prefix = 'gs://example-data-lake-curated/equity-prices/', require_hive_partition_filter = TRUE -- forces callers to filter on partition columns);
Query the external table with partition pruning
-- This scans ONLY year=2026/month=03/day=22/ — not the full tableSELECT instrument_id, close, volumeFROM `example-project.lake_curated.equity_prices_ext`WHERE year = 2026 AND month = 3 AND day = 22 AND instrument_id IN ('MSFT', 'AAPL', 'GOOG');
Materialize Hot External Tables
External tables are re-read from GCS on every query. For tables queried many times per day, materialize them into native BigQuery tables nightly:
CREATE OR REPLACE TABLE `example-project.warehouse.equity_prices`PARTITION BY trade_dateCLUSTER BY instrument_idAS SELECT * FROM `example-project.lake_curated.equity_prices_ext`WHERE year = EXTRACT(YEAR FROM CURRENT_DATE());
Native tables benefit from BigQuery’s statistics, caching, and slot-based optimization.
Dataproc for Heavy Transformation
For complex transformations that exceed what SQL can express efficiently (e.g., time-series gap-filling across 50,000 instruments, ML feature engineering, joining 10+ data sources), use Dataproc (managed Spark) to read from GCS, transform, and write back.
GCS charges a minimum storage duration for NEARLINE (30 days) and COLDLINE (90 days). If you delete a COLDLINE object after 10 days, you are charged for 90 days. Design lifecycle transitions so objects have lived in the current class for at least the minimum duration before transitioning or deleting. See gcs-buckets-and-lifecycle for the full cost model.
Safe Lifecycle Transition Design
Structure the lifecycle rule chain so each transition fires only after the object has fully served its minimum duration in the current class: STANDARD for the first 30 days, transition to NEARLINE at day 30 (not before), transition to COLDLINE at day 90, and delete at day 180 (or retain permanently in COLDLINE for the cleansed zone). This avoids minimum-duration charges and aligns transitions with natural access patterns — recently ingested data is accessed more frequently.
Columnar Compression Efficiency
Switching from CSV to Parquet with Snappy compression typically reduces storage by 60–80%. This directly reduces BigQuery on-demand scan costs (priced per byte scanned) and GCS storage costs.
Source format
Rows
Raw size
Parquet + Snappy
Reduction
CSV (daily equity prices)
5M
800 MB
120 MB
85%
JSON (API responses)
100K
200 MB
15 MB
92%
CSV (order book)
50M
8 GB
900 MB
89%
Partition Elimination
A well-partitioned lake reduces BigQuery scan bytes in proportion to how selective the partition filter is. A query filtering on a single day from a 3-year daily-partitioned table scans approximately 1/1095 of the data.
Verify partition elimination is working:
# BigQuery dry run with partition filter — check bytes processedbq query --use_legacy_sql=false --dry_run 'SELECT instrument_id, closeFROM `example-project.lake_curated.equity_prices_ext`WHERE year = 2026 AND month = 3 AND day = 22'# "This query will process X bytes when run."# X should equal ~1 day of data, not the full table
Data Lake Governance Checklist
Before treating a data lake zone as production-ready:
Every dataset has a catalog entry (name, owner, schema, location, refresh cadence)
Every dataset has a lineage record (upstream sources, producing pipeline, downstream consumers)
PII columns are identified and masked/tokenized at the cleansed zone boundary
Retention policies are defined and implemented as lifecycle rules on each bucket
Access control is enforced at the bucket level (not project level) per zone
Schema validation runs at the landing → cleansed transition (bad records quarantined)
File sizes are within the 128–512 MB optimal range (compaction schedule exists)
Idempotent writes — re-running any pipeline for a date produces the same output
Partition pruning verified with dry-run queries (external tables only scan needed partitions)
Naming conventions followed: lowercase, underscores, no spaces, Hive-style paths