03 - Lakehouse Architecture

20 min read

Lakehouse Architecture

Quote

“A lakehouse is a data management system based on low-cost storage that also provides traditional analytical DBMS management and performance features.”

— Michael Armbrust (co-creator of Delta Lake)

Why the Lakehouse Emerged

To understand the lakehouse you need to understand what it replaced — and why both predecessors failed in isolation.

The Data Lake Problem (2010–2018)

The original data lake premise was compelling: store all raw data cheaply on object storage (HDFS, then S3/GCS) and process it with schema-on-read. No ETL, no upfront modeling, maximum flexibility. In practice, data lakes became “data swamps”:

  • No ACID guarantees. A failed pipeline write left partial or corrupt files with no rollback mechanism.
  • No schema enforcement. Any producer could write any schema. Downstream consumers broke silently when upstream schema changed.
  • No time travel. “What did this table look like last Tuesday?” — impossible without maintaining your own snapshot infrastructure.
  • No fine-grained updates. Correcting a single row meant rewriting entire Parquet partitions.
  • Governance collapse. Without metadata, datasets became undiscoverable, untrustworthy, and unusable.
  • Massive small-files problem. Streaming writes produced millions of tiny files, making queries scan thousands of objects for no data benefit.

The Data Swamp Anti-Pattern

A data lake without governance is a liability, not an asset. Teams that dumped data into S3 “to process later” typically found that “later” never came, the schema was undocumented, and the cost of making the data usable exceeded the cost of re-extracting from source.

Safe Pattern: Govern from the First Byte

Apply an open table format (Iceberg or Delta Lake) from the very first write — even to raw/Bronze tables. Register all tables in a catalog (BigLake Metastore, AWS Glue, Nessie) at creation time, enforce schema-on-write, and document grain and ownership before any downstream pipeline reads the data. Governance applied retroactively is orders of magnitude more expensive than governance applied at ingestion.

The Data Warehouse Problem (2000–present)

Data warehouses (Snowflake, Redshift, BigQuery, SQL Server) solved governance but imposed hard constraints:

  • Proprietary storage formats. Data inside a Snowflake table is not accessible by Spark or DuckDB without going through Snowflake’s compute (and billing).
  • Cost at scale. Storing every raw event in a warehouse at warehouse pricing is 5–20x more expensive than object storage.
  • ML/AI friction. Training ML models requires raw, granular data in formats that Python and Spark consume natively (Parquet, TFRecord). Exporting from a warehouse for every training run is slow and expensive.
  • Semi-structured data limitations. Warehouses can store JSON but were not designed to query deeply nested, high-cardinality event streams efficiently.

The Lakehouse Synthesis

The lakehouse resolves both sets of problems by separating storage from compute and adding a transaction layer on top of open formats:

flowchart TD
    engines["QUERY ENGINES<br/>Spark | Trino | Presto | DuckDB | BigQuery | Athena"]
    format["OPEN TABLE FORMAT LAYER<br/>Delta Lake | Apache Iceberg | Apache Hudi<br/>Metadata: catalogs, manifests, snapshots, txn log"]
    storage["OBJECT STORAGE<br/>Google Cloud Storage | Amazon S3 | ADLS Gen2<br/>Parquet files — cheap, durable, open"]

    engines -->|"reads/writes via<br/>open table format API"| format
    format -->|"physical files"| storage

    style engines fill:#1a1a2e,stroke:#22d3ee,color:#fff
    style format fill:#1a1a2e,stroke:#bb9af7,color:#fff
    style storage fill:#1a1a2e,stroke:#9ece6a,color:#fff

The table format layer is what makes the lakehouse work. It is a metadata contract on top of files — any engine that implements the format spec can read and write the same data, with full ACID semantics.

Key Properties of a Lakehouse

ACID Transactions on Object Storage

Object storage (GCS, S3) has no native transaction support — it is a key-value store for blobs. Open table formats implement ACID by managing a transaction log that tracks every operation:

  • Atomicity: a write either commits all files or none. A failed job leaves no partial state visible to readers.
  • Consistency: readers always see a consistent snapshot; a write in progress does not affect concurrent reads.
  • Isolation: multiple writers can operate concurrently using optimistic concurrency control; conflicts are detected and the losing writer retries.
  • Durability: once committed to the transaction log (which is itself written to object storage), a write is permanent.

How Delta Lake Achieves Atomicity

Delta Lake writes new Parquet files to the storage location, then atomically updates a _delta_log/ transaction log entry. Readers query the log first to determine which files constitute the current table state. Files not referenced in the log are invisible — partial writes simply never appear in the log.

Schema Enforcement and Evolution

Unlike a raw data lake, a lakehouse table has a defined schema that is enforced on write:

  • Enforcement: writing a dataframe with an incompatible schema raises an error rather than silently producing bad data.
  • Evolution: new columns can be added safely without breaking existing readers. Column types can be widened (INT → LONG) but not narrowed.
  • Partition evolution (Iceberg): the partition scheme of a table can change without rewriting historical data — Iceberg tracks the partition spec per snapshot.

Time Travel

Every write to a lakehouse table creates a new snapshot. Snapshots are retained according to a configurable retention policy. This enables:

# Delta Lake time travel — Spark
df = spark.read.format("delta") \
    .option("timestampAsOf", "2026-01-01") \
    .load("gs://my-bucket/events")
 
# Iceberg time travel — SQL via Spark SQL
spark.sql("""
SELECT * FROM catalog.db.events
TIMESTAMP AS OF '2026-01-01 00:00:00'
""")
 
# Query a specific snapshot ID
spark.sql("""
SELECT * FROM catalog.db.events
VERSION AS OF 42
""")

Time travel is critical for:

  • Auditing: regulatory requirements to produce data as it existed at a specific point in time.
  • ML reproducibility: training a model on the exact data snapshot that was available at training time.
  • Debugging: comparing a pipeline output before and after a change.
  • GDPR right-to-erasure rollbacks: verifying a deletion propagated correctly across historical snapshots.

Row-Level Updates and Deletes

Traditional Parquet on object storage cannot update or delete individual rows — you must rewrite entire partition files. The lakehouse formats handle row-level mutations:

  • Copy-on-Write (CoW): on UPDATE/DELETE, the affected Parquet files are rewritten with the changes applied. Reads are fast (no merging needed); writes are expensive for large files.
  • Merge-on-Read (MoR): changes are written as small delta files (delete vectors, row-group updates). Reads merge the base files with the deltas on the fly; writes are fast but reads do more work.

Most production lakehouses use Copy-on-Write for dimension tables (slow-changing, query-heavy) and Merge-on-Read for fact tables (high-write, streaming).

Open Table Formats Comparison

For a deep technical dive into each format’s metadata model, see open-table-formats. The summary comparison:

DimensionDelta LakeApache IcebergApache Hudi
OriginDatabricks (2019)Netflix (2020, Apache)Uber (2019, Apache)
Metadata modelTransaction log JSON filesManifest + snapshot treeTimeline + log compaction
Partition evolutionLimited (requires rewrite)Full — per-snapshot partition specLimited
Hidden partitioningNoYes — engine computes partition from column valuesNo
Row-level deletesDelete vectors (v2+)Equality delete filesDelete log (MoR)
Streaming supportStrong (Spark Structured Streaming)Strong (Flink, Spark)Native streaming-first design
Catalog supportDatabricks Unity Catalog, HMS, AWS GlueNessie, Polaris, Tabular, HMS, AWS Glue, BigLakeHMS, AWS Glue
GCP / BigQuery integrationVia Dataproc, not native BigQueryBigQuery Iceberg (GA), BigLakeLimited
Primary ecosystemDatabricksMulti-vendor, openHudi ecosystem
Best forDatabricks shopsMulti-engine, cloud-agnosticStreaming-heavy CDC workloads

Choosing a Format for GCP

On Google Cloud Platform, Apache Iceberg is the strategic choice. BigQuery has native Iceberg support (BigQuery Iceberg tables), BigLake Metastore is the managed catalog, and Dataproc/Spark reads Iceberg natively. Delta Lake works on Dataproc but has no native BigQuery integration. Hudi is rarely seen in GCP-first environments.

Medallion Architecture as the Lakehouse Pattern

The medallion-architecture (bronze / silver / gold) is the canonical organizational pattern for data within a lakehouse. Each layer is a set of lakehouse tables (Iceberg or Delta) in object storage, with increasing quality and decreasing granularity:

flowchart LR
    src["Raw Sources<br/>APIs, DBs, Files"]
    bronze["Bronze Layer<br/>Raw, schema-on-write<br/>Iceberg / Delta on GCS"]
    silver["Silver Layer<br/>Cleaned, deduplicated<br/>Iceberg / Delta on GCS"]
    gold["Gold Layer<br/>Aggregated, business-ready<br/>Iceberg / Delta on GCS"]

    src --> bronze --> silver --> gold

    style src fill:#1a1a2e,stroke:#7aa2f7,color:#fff
    style bronze fill:#1a1a2e,stroke:#e0af68,color:#fff
    style silver fill:#1a1a2e,stroke:#bb9af7,color:#fff
    style gold fill:#1a1a2e,stroke:#9ece6a,color:#fff

Key differences from a purely SQL-Server-based medallion implementation:

  • Bronze tables use Merge-on-Read — streaming writes arrive constantly, CoW is too expensive.
  • Silver tables use Copy-on-Write — quality transformations are batch, reads are frequent.
  • Gold tables may be materialized views in BigQuery for fast BI access, pointing at the Iceberg silver tables via BigLake.

The dbt-transformation-layer can manage the silver → gold transformations using incremental models against Iceberg tables via Spark or Trino.

GCP Lakehouse: BigLake and Unity Catalog

BigLake (GCP’s Lakehouse Management Layer)

BigLake is Google’s lakehouse governance layer. It consists of:

  • BigLake Metastore: a managed catalog compatible with the Apache Hive Metastore (HMS) API and the Iceberg REST Catalog spec. Spark, Flink, and Hive can all point at it.
  • BigLake tables: a table type in BigQuery that reads data from GCS (Parquet, Iceberg, Delta, ORC, Avro) without copying it. Storage stays in GCS; BigQuery provides the SQL query engine.
  • Fine-grained access control: row-level security and column-level masking policies are enforced at the BigLake layer — the same policy applies whether a user queries via BigQuery SQL or via Spark.
  • Data Boost: a serverless read path that lets Bigtable and Spanner data be queried without consuming database compute.

GCP Lakehouse Stack

flowchart BT
    write["Dataflow / Dataproc / Spark<br/>Writes new data"]
    gcs["Google Cloud Storage<br/>Parquet + Iceberg metadata"]
    catalog["BigLake Metastore<br/>Catalog: databases, tables, schemas"]
    bq["BigQuery<br/>SQL queries, BI"]

    write --> gcs
    gcs --> catalog
    catalog -->|"BigLake tables<br/>(external tables pointing at GCS)"| bq

    style write fill:#1a1a2e,stroke:#9ece6a,color:#fff
    style gcs fill:#1a1a2e,stroke:#e0af68,color:#fff
    style catalog fill:#1a1a2e,stroke:#bb9af7,color:#fff
    style bq fill:#1a1a2e,stroke:#22d3ee,color:#fff

Python: create a BigLake Iceberg table via BigQuery client

from google.cloud import bigquery
 
client = bigquery.Client(project="my-project")
 
# Create a BigQuery dataset that maps to a BigLake catalog
external_config = bigquery.ExternalConfig("ICEBERG")
external_config.source_uris = ["gs://my-bucket/warehouse/events/"]
 
table = bigquery.Table("my-project.my_dataset.events")
table.external_data_configuration = external_config
 
client.create_table(table)
print("BigLake Iceberg table created")

Unity Catalog (Databricks)

Unity Catalog is Databricks’ governance layer for the Databricks Lakehouse:

  • Unified catalog for Delta Lake tables, files, ML models, and dashboards.
  • Fine-grained access control (row filters, column masks) enforced at the catalog level.
  • Lineage tracking: every read and write is recorded, enabling column-level lineage.
  • Delta Sharing: open protocol for sharing Delta/Iceberg tables with external parties without copying data.

Tabular (Iceberg-native)

Tabular is the company founded by the Apache Iceberg creators. It provides:

  • A managed Iceberg catalog (REST catalog spec).
  • Table optimization services (compaction, clustering, expiration) as a managed service.
  • Engine-agnostic: works with Spark, Trino, Flink, DuckDB, Snowflake.

Query Engines for the Lakehouse

A key lakehouse advantage is engine independence — the same data can be queried by multiple engines:

EngineBest ForNotes
Apache SparkLarge-scale batch ETL, ML feature engineeringThe dominant lakehouse write engine; native Delta and Iceberg support
Trino / PrestoInteractive ad-hoc SQL queriesSub-second latency at petabyte scale; excellent Iceberg support
DuckDBLocal/single-node analytics, fast development iterationCan read Parquet and Iceberg directly; no cluster needed
BigQueryBI queries, dashboards, SQL familiarityNative Iceberg support via BigLake tables; serverless pricing
Apache FlinkStreaming ETL writing to lakehouse tablesNative Iceberg sink; low-latency streaming with exactly-once
ksqlDBKafka-integrated streaming SQLPairs with Kafka for event-driven lakehouse ingestion

DuckDB for Local Development

DuckDB is the fastest way to develop and test lakehouse queries locally. It reads Parquet files from GCS directly (with INSTALL httpfs; LOAD httpfs; SET s3_region='auto') and supports basic Iceberg catalog queries. Use it to prototype transformations before scaling to Spark.

DuckDB reading Parquet from GCS

-- DuckDB local query against GCS Parquet files
INSTALL httpfs;
LOAD httpfs;
 
-- Set GCP credentials
SET gcs_access_key_id = 'GOOG...',
    gcs_secret_access_key = '...';
 
SELECT
    event_date,
    COUNT(*) AS event_count,
    SUM(amount) AS total_amount
FROM read_parquet('gs://my-bucket/warehouse/events/**/*.parquet')
WHERE event_date >= '2026-01-01'
GROUP BY event_date
ORDER BY event_date;

Lakehouse vs Data Warehouse vs Data Lake

DimensionData LakeData WarehouseLakehouse
Storage formatOpen (Parquet, JSON, CSV, raw)Proprietary (Snowflake micropartitions, BQ Capacitor)Open (Parquet + table format metadata)
Storage costVery low (GCS/S3 pricing)High (warehouse pricing includes storage)Very low (GCS/S3 pricing)
ACID transactionsNoneFullFull (via Delta/Iceberg/Hudi)
Schema enforcementNone (schema on read)StrictEnforced on write
Time travelNone (unless you build it)Limited (Snowflake Time Travel, BQ snapshots)Native (every format)
SQL query supportVia external engines onlyNative, optimizedVia multiple engines
ML/AI accessDirect (Spark/Python reads files natively)Export required (expensive, slow)Direct (same files, governed)
Streaming ingestionYes (just write files)Limited, expensiveYes (MoR, streaming formats)
Governance & securityNoneStrongStrong (catalog + ACLs)
Vendor lock-inLowHighLow (open formats)
Compute engine choiceAnyWarehouse-specificAny (Spark, Trino, DuckDB, BigQuery)
Update/delete rowsNoYesYes (CoW or MoR)
Typical usersData scientists, ML engineersData analysts, BIAll of the above

When to Choose the Lakehouse

Choose the lakehouse when

  • You have multiple compute engines that must access the same data (Spark for ETL, BigQuery for BI, Python for ML).
  • Your data volume makes warehouse storage pricing prohibitive (>10 TB active data).
  • You need ML/AI training on raw or semi-raw data at scale.
  • Your regulatory environment requires long-retention data with point-in-time query capability.
  • You are building a multi-tenant data platform where different teams use different tools.
  • You are managing semi-structured or unstructured data alongside structured.
  • You need to implement data mesh — domain teams owning their data products in open formats that any consumer can read.

Do not choose the lakehouse when

  • Your team is small and primarily doing SQL-based BI — a managed warehouse (BigQuery, Snowflake) has far less operational overhead.
  • You are in early-stage product development where iteration speed matters more than scalability.
  • You do not have Spark/Trino expertise — the lakehouse requires engineering investment to operate correctly.
  • Your primary use case is OLTP (operational transactions) — a lakehouse is an analytics platform, not a database.

Operational Complexity

A lakehouse is a distributed system. You are now responsible for compaction (merging small files into large ones), snapshot expiration (cleaning up old table versions), catalog management, and compute cluster sizing. Managed services (Databricks, Tabular, Google Dataproc Metastore) reduce this burden but do not eliminate it. Budget for operational engineering from day one.

Safe Pattern: Automate Maintenance from Day One

Schedule compaction, snapshot expiration, and orphan-file removal as recurring jobs in your orchestrator (Airflow, Databricks Workflows) at launch — not after performance degrades. Start with conservative defaults (compact daily, expire snapshots after 7 days, remove orphans weekly) and tune based on observed file-size distributions and query latency metrics.

Real-World Implementations

Databricks Lakehouse Platform

Databricks is the company that invented Delta Lake and coined the term “lakehouse.” Their managed platform provides:

  • Managed Delta Lake: automatic compaction, Z-ordering (multi-column clustering), Liquid Clustering (adaptive partitioning).
  • Unity Catalog: governance, lineage, row-level security.
  • Databricks SQL: SQL warehouse for BI on Delta tables with query result caching.
  • MLflow: integrated ML experiment tracking and model registry.
  • Delta Live Tables: declarative pipeline framework for building medallion architecture with automatic dependency resolution and quality assertions.

Databricks Delta Live Tables (declarative medallion)

import dlt
from pyspark.sql.functions import col, current_timestamp
 
# Bronze: raw ingestion — append from streaming source
@dlt.table(
    name="events_bronze",
    comment="Raw events from Pub/Sub — no cleaning, no deduplication"
)
def events_bronze():
    return (
        spark.readStream
        .format("cloudFiles")  # Auto Loader — GCS file discovery
        .option("cloudFiles.format", "json")
        .load("gs://my-bucket/raw/events/")
    )
 
# Silver: cleaned, deduplicated
@dlt.table(
    name="events_silver",
    comment="Cleaned events — nulls removed, duplicates eliminated"
)
@dlt.expect_or_drop("valid_event_id", "event_id IS NOT NULL")
@dlt.expect_or_drop("valid_amount", "amount > 0")
def events_silver():
    return (
        dlt.read_stream("events_bronze")
        .dropDuplicates(["event_id"])
        .withColumn("ingested_at", current_timestamp())
    )
 
# Gold: business aggregate
@dlt.table(
    name="daily_revenue_gold",
    comment="Daily revenue rollup — dashboard-ready"
)
def daily_revenue_gold():
    return (
        dlt.read("events_silver")
        .groupBy("event_date", "product_id")
        .agg({"amount": "sum", "event_id": "count"})
    )

Snowflake with Iceberg Support

Snowflake added Apache Iceberg table support — Snowflake can serve as the catalog and query engine for Iceberg tables stored in your own cloud storage bucket:

  • Snowflake-managed Iceberg: Snowflake manages the metadata; you own the storage.
  • External Iceberg: tables created by Spark or another engine, registered in Snowflake as external Iceberg tables.
  • This enables “bring your own storage” while leveraging Snowflake’s SQL engine and BI ecosystem.

BigQuery with BigLake

BigQuery’s lakehouse story is built around:

  • BigLake tables: read Iceberg, Delta, Parquet, ORC from GCS with BigQuery SQL — no data copy.
  • Omni: run BigQuery SQL against data in AWS S3 or Azure ADLS.
  • Materialized views on external tables: BigQuery can materialize the results of queries against BigLake Iceberg tables, providing sub-second BI latency.

BigQuery query against a BigLake Iceberg table

-- BigQuery SQL — reads directly from GCS Iceberg table
SELECT
  event_date,
  product_category,
  SUM(revenue) AS total_revenue,
  COUNT(DISTINCT user_id) AS unique_users
FROM `my-project.my_dataset.events_silver`  -- BigLake Iceberg table
WHERE event_date BETWEEN '2026-01-01' AND '2026-03-22'
GROUP BY event_date, product_category
ORDER BY event_date, total_revenue DESC;

For BigQuery query optimization on external tables, see querying-and-cost-optimization.

Compaction and Table Maintenance

A critical operational responsibility in any lakehouse is table maintenance. Streaming writes and frequent small updates create many small Parquet files, which degrade read performance:

Iceberg table maintenance with Spark

from pyspark.sql import SparkSession
 
spark = SparkSession.builder \
    .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \
    .config("spark.sql.catalog.glue", "org.apache.iceberg.spark.SparkCatalog") \
    .getOrCreate()
 
# Compact small files into target file size (512 MB)
spark.sql("""
CALL glue.system.rewrite_data_files(
  table => 'db.events',
  options => map(
    'target-file-size-bytes', '536870912',
    'min-file-size-bytes',    '134217728'
  )
)
""")
 
# Expire old snapshots (keep 7 days)
spark.sql("""
CALL glue.system.expire_snapshots(
  table => 'db.events',
  older_than => TIMESTAMP '2026-03-15 00:00:00',
  retain_last => 10
)
""")
 
# Remove orphan files (files not in any snapshot)
spark.sql("""
CALL glue.system.remove_orphan_files(
  table => 'db.events',
  older_than => TIMESTAMP '2026-03-15 00:00:00'
)
""")

Automate Maintenance with Airflow

Schedule Iceberg maintenance jobs as daily Airflow DAGs. Run compaction after the nightly batch load, expire snapshots weekly, and remove orphans monthly. Failing to do this will progressively degrade query performance and inflate storage costs.

Connection to Streaming Architecture

The lakehouse is primarily a batch analytics architecture, but it increasingly handles streaming workloads. For streaming pipelines writing to lakehouse tables, see streaming-architecture:

  • Apache Flink writes to Iceberg tables with exactly-once semantics via the Iceberg Flink sink.
  • Spark Structured Streaming writes to Delta Lake with micro-batch or continuous processing.
  • Pub/Sub → Dataflow (Apache Beam) → Iceberg on GCS is the canonical GCP streaming-to-lakehouse path.

Data Architecture Lakehouse Architecture References

Graph View

Backlinks