04 - Data Mesh Architecture

18 min read

Data Mesh Architecture

Quote

“There is accidental complexity: the complexity that we — engineers, architects, and designers — create in our solutions. Accidental complexity can and should be reduced.”

— Zhamak Dehghani, Data Mesh (2022)

The Four Principles

Dehghani’s data mesh framework rests on four principles. Every implementation must address all four — missing any one produces an incomplete, fragile system.

Principle 1: Domain-Oriented Ownership

What it means: The team that creates the data owns it — including its quality, schema, documentation, freshness, and access control. The payments domain team owns payments data. The orders team owns orders data. They are accountable for the data product SLA from source system to downstream consumer.

Why it matters: In a centralized model, the data team ingests data from 40 source systems they do not understand deeply. They cannot detect that the payments API started sending amounts in cents instead of dollars. The payments engineering team would catch that in 30 seconds. Domain ownership moves the accountability where the knowledge lives.

What it requires in practice

  • Domain teams must accept that data is part of their product — not someone else’s problem.
  • Data engineers may be embedded in domains rather than centralized.
  • SLAs for data quality (freshness, completeness, schema stability) are owned at the domain level.
  • The organizational chart must support this — you cannot have domain ownership without domain authority.

Example mapping

Domain Team            Owns
─────────────────────────────────────────────────────────────
Payments               payments_events, refunds, chargebacks
Orders                 orders, order_items, fulfillment_events
Customer               customer_profiles, session_events
Risk                   fraud_signals, risk_scores, denials
Finance                ledger_entries, reconciliation_summaries

The Hardest Principle

Domain ownership is the hardest principle to implement because it requires engineering teams to take on responsibilities they currently hand off: data quality monitoring, schema documentation, consumer SLA management. This is a culture change, not a technical one. Executive sponsorship and clear incentive structures are prerequisites.

Making Domain Ownership Succeed

Start with one high-value, willing domain team as a pilot. Provide an embedded data engineer for the first two sprints to build the tooling alongside the domain team, not for them. Define the ownership contract concretely: the domain team is responsible for freshness SLA, schema documentation, and a runbook for data quality incidents. Celebrate the first successful consumer who unblocked themselves without filing a ticket to the central team — that is the proof point that changes the culture.

Principle 2: Data as a Product

What it means: Domain data is not an internal byproduct of operational systems — it is a product, designed for consumption by others. A data product has defined consumers, a contract (schema + SLA), and is actively maintained and improved.

The six characteristics of a data product (Dehghani)

CharacteristicWhat it meansExample
DiscoverableListed in a catalog; searchable by consumersDataHub entry with description, owner, schema, sample data
AddressableHas a stable, unique address consumers use to access itdata-product://payments.com/transactions/v2 or a dataset URI
TrustworthyHas guaranteed freshness, completeness, and accuracy SLAs”Updated within 15 minutes of source event, 99.9% completeness”
Self-describingSchema is embedded or linked; no tribal knowledge requiredAvro/Protobuf schema in registry; BigQuery table description
InteroperableUses standard formats and access patterns any consumer can useIceberg table on GCS; BigQuery dataset; Kafka topic with Avro
SecureAccess controls are policy-driven; no ad-hoc permission grantsIAM bindings; column-level security; row-level security

A data product is not

  • A raw operational database dump with no documentation.
  • A CSV file emailed to a stakeholder.
  • A pipeline that “should be fine” without a freshness SLA.
  • A table whose schema changes without notice to consumers.

Data product ownership boundary

Source System                    Data Product                    Consumers
(Payments API)                   (owned by Payments team)        (Finance, Risk, Analytics)
      │                                │                               │
      ▼                                ▼                               ▼
payments_raw_events  ──►  payments_transactions_v2  ──────►  downstream uses
(internal impl)           (public contract, SLA,              (they depend on the
                           schema in registry)                 contract, not impl)

Principle 3: Self-Serve Data Platform

What it means: A central platform team builds the infrastructure and tooling that domain teams use to build, deploy, and operate data products — without needing platform team involvement for each data product. The platform is a product. Its customers are domain teams.

What the platform provides

  • Storage infrastructure: managed object storage (GCS buckets), managed catalogs (BigLake Metastore, DataHub).
  • Compute infrastructure: managed Spark clusters, Dataflow runners, BigQuery slots.
  • Schema registry: Confluent Schema Registry or equivalent — schema registration, compatibility checking, evolution rules.
  • Data catalog: DataHub, Amundsen, or OpenMetadata — automatic schema discovery, lineage, ownership tagging.
  • Pipeline framework: standardized templates for data product pipelines (Terraform modules for GCS + IAM, Airflow DAG templates, dbt project scaffolding).
  • Observability: standard metrics (freshness, row count, null rate) automatically collected for every data product.
  • Access request UI: self-service access requests with automated approval workflows, not a Jira ticket to the platform team.

Platform Team Anti-Pattern

A platform team that processes individual requests (“please create a bucket for us”, “please add this IAM binding”) is not a self-serve platform — it is a centralized bottleneck with extra steps. The measure of success is: can a domain team launch a new data product without filing a single ticket to the platform team?

Example self-serve platform tooling stack

Domain Team Experience          Platform Layer                  Infrastructure
─────────────────────────────────────────────────────────────────────────────
git push → CI deploys           dbt Cloud / Airflow DAG         GCP Dataproc
  schema to registry             Template Library                Pub/Sub
  → DataHub auto-registers       Terraform Module Registry       BigQuery
  → freshness monitor starts     Schema Registry (Confluent)     BigLake Metastore
  → access policy applied        DataHub / OpenMetadata          Cloud IAM
                                 Datadog Integration             GCS

Principle 4: Federated Computational Governance

What it means: Global governance standards (data classification, retention policies, compliance rules, interoperability contracts) are defined centrally but enforced automatically by the platform — not by a central team reviewing every change. Domain teams operate autonomously within these guardrails.

What federated governance covers

  • Data classification: every dataset is tagged with sensitivity level (public, internal, confidential, restricted). The tagging rule is global; the tagging is done by domain teams.
  • Retention policies: automated enforcement — a “confidential” tag automatically triggers a 90-day lifecycle rule on the GCS bucket.
  • Schema compatibility: the schema registry enforces backward/forward compatibility rules globally — a domain team cannot publish a breaking schema change without explicit versioning.
  • Access policies: standard IAM role templates (data-product-reader, data-product-writer) are defined globally and applied per data product. Ad-hoc permissions are not allowed.
  • Data quality SLAs: minimum quality standards (freshness, non-null rate on key columns) are enforced by automated monitors; failures trigger SLA breach alerts.
  • Interoperability standards: all data products must use one of the approved formats (Iceberg on GCS, BigQuery dataset, Kafka + Avro topic) so any consumer can connect without bespoke integration.

The key insight: computational governance. The rules are code, not process. They are enforced by the platform at deploy time, not by a governance committee at review time. A domain team that tries to deploy a data product without a schema registration fails CI — they cannot deploy without complying.

Data Products in Practice

Data Contract

A data contract is the formal, versioned agreement between a data product owner and its consumers. It specifies:

# data-contract.yaml — payments_transactions_v2
version: "2.1.0"
name: payments_transactions
domain: payments
owner: payments-platform-team@company.com
description: "All completed payment transactions with enriched metadata"
 
schema:
  type: avro
  registry_url: "https://schema-registry.company.com"
  subject: "payments.transactions.v2-value"
 
sla:
  freshness_max_lag_minutes: 15
  completeness_min_percent: 99.9
  availability_percent: 99.5
 
quality_assertions:
  - column: transaction_id
    rule: not_null
    critical: true
  - column: amount_usd
    rule: "amount_usd > 0"
    critical: true
  - column: status
    rule: "status IN ('completed', 'failed', 'pending')"
    critical: false
 
access:
  classification: confidential
  approved_consumers:
    - finance-domain
    - risk-domain
    - analytics-platform
  request_url: "https://data-catalog.company.com/access/payments_transactions"
 
changelog:
  - version: "2.1.0"
    date: "2026-03-01"
    change: "Added `processor_id` column (backward compatible)"
  - version: "2.0.0"
    date: "2025-11-01"
    change: "Renamed `amount` to `amount_usd`; breaking change — migration guide in repo"

Data contracts are validated in CI. A change to the payments pipeline that would violate the contract (e.g., dropping a non-null column) fails the CI build before it reaches production.

Schema Registry

A schema registry stores and versions the schemas for data products. It enforces compatibility rules:

  • BACKWARD compatibility: new schema can read data written with old schema. Old consumer code still works.
  • FORWARD compatibility: old schema can read data written with new schema. New producer, old consumer.
  • FULL compatibility: both backward and forward. The strictest and safest for long-lived data products.

Registering a schema with Confluent Schema Registry (Python)

from confluent_kafka.schema_registry import SchemaRegistryClient, Schema
 
sr_conf = {"url": "https://schema-registry.company.com"}
sr_client = SchemaRegistryClient(sr_conf)
 
avro_schema_str = """
{
  "type": "record",
  "name": "PaymentTransaction",
  "namespace": "com.company.payments",
  "fields": [
    {"name": "transaction_id", "type": "string"},
    {"name": "amount_usd",     "type": "double"},
    {"name": "currency",       "type": "string"},
    {"name": "status",         "type": "string"},
    {"name": "processor_id",   "type": ["null", "string"], "default": null}
  ]
}
"""
 
schema = Schema(avro_schema_str, schema_type="AVRO")
schema_id = sr_client.register_schema(
    subject_name="payments.transactions.v2-value",
    schema=schema
)
print(f"Registered schema ID: {schema_id}")
 
# Check compatibility before pushing a change
compatibility = sr_client.test_compatibility(
    subject_name="payments.transactions.v2-value",
    schema=new_schema
)
print(f"Compatible: {compatibility}")

Data Mesh vs Centralized Data Team

DimensionCentralized Data TeamData Mesh
Data ownershipCentral team owns all pipelines and datasetsDomain team owns its data products end-to-end
BottleneckCentral team is bottleneck for every requestDomain teams operate autonomously
Domain knowledgeCentral team learns domain graduallyDomain team has deep, native domain knowledge
ScalingScales linearly with team size — expensiveScales with number of domain teams
ConsistencyHigh — one team enforces standardsRequires federated governance to maintain consistency
Quality accountabilityCentral team blamed for data qualityDomain team accountable for their data product SLA
Onboarding costLow — one team to learnHigh — every domain team needs data engineering capability
Best forSmall-medium orgs, early-stage, <20 data sourcesLarge orgs, many domains, >50 data sources, rapid growth
RiskBottleneck kills velocityWithout governance, becomes distributed data swamp

When Centralized Still Wins

Data mesh is not inherently better than centralized. For an organization with 5 engineers and 10 data sources, a centralized data team with a well-run medallion architecture delivers more value with less complexity. Data mesh is an answer to organizational scale problems, not a universally superior architecture.

Technology Enablers

Data mesh is technology-agnostic, but certain technologies make it practical:

Kafka (Message Broker)

Pub/Sub and Kafka are the canonical event backbone for data mesh. Domain teams publish domain events to their own topics; consumers subscribe. Decoupling is physical — consumers do not call the domain’s API; they read from the event stream.

Payments Domain          Kafka Broker          Consumers
─────────────────────────────────────────────────────────
payment_service  ──►  payments.transactions  ──►  Finance analytics
                       (payments team owns)   ──►  Risk scoring
                                             ──►  Reconciliation

Schema Registry (Confluent / AWS Glue / Apicurio)

Stores and versions Avro/Protobuf/JSON Schema schemas. Enforces compatibility rules automatically. Enables consumers to deserialize data without out-of-band schema sharing.

Data Catalogs

ToolKey Strengths
DataHub (LinkedIn, open source)Strong lineage, GraphQL API, rich plugin ecosystem, GCP integration
Amundsen (Lyft, open source)Search-first UX, good for large orgs already using Lyft stack
OpenMetadataOpen standard API, comprehensive metadata model, active community
Google DataplexNative GCP, auto-discovery for BigQuery and GCS, policy enforcement
Collibra / AlationEnterprise features, data governance workflows, expensive

A data catalog in a data mesh context must support:

  • Automatic schema discovery (pulls schemas from schema registry and tables).
  • Ownership tagging (maps datasets to domain teams).
  • Lineage tracking (shows which data products depend on which upstream products).
  • Access request workflows (self-serve, not ticket-based).
  • Data quality metric display (freshness, completeness, SLA status).

Open Table Formats as Data Product Storage

Apache Iceberg tables on object storage are the natural storage layer for data products in a mesh:

  • Engine-independent: any domain’s consumers can read the table with Spark, DuckDB, BigQuery, or Trino.
  • Schema is embedded in the table metadata — self-describing.
  • Access control is at the catalog and storage level, not embedded in any single compute engine.

When Data Mesh Makes Sense

Strong indicators for data mesh

  • Your organization has 10+ distinct business domains, each with their own engineering teams.
  • The central data team is a chronic bottleneck — time-to-data for new datasets is measured in weeks.
  • Data quality issues are repeatedly traced back to the central team not understanding the source domain.
  • You have active regulatory requirements that mandate clear data ownership and lineage.
  • Your engineering org already practices domain-driven design (DDD) in application development.

Indicators data mesh is the wrong choice

  • Your organization has fewer than 50 engineers total.
  • Domain teams do not have data engineering capacity and are unwilling to build it.
  • You are in a highly regulated industry with strict, centralized compliance requirements (some financial services contexts).
  • You are still trying to get basic data infrastructure working — data mesh requires mature foundations.
  • Your leadership is looking for a technology solution to what is fundamentally an organizational problem.

Common Pitfalls

Treating It as a Technology Problem

Teams buy a data catalog, implement Kafka, and call it “data mesh.” Without domain ownership and governance, they now have a distributed data swamp with better tooling.

No Governance — Distributed Data Anarchy

Removing the central team without federated governance produces chaos: inconsistent schemas, no discoverability, incompatible formats, no lineage. Governance must be designed before domains are granted autonomy.

Duplication Without Coordination

Domain teams independently build the same data products because there is no discovery mechanism. A data catalog with clear ownership is mandatory from day one to prevent this.

Underestimating Organizational Change

The technology takes weeks to implement. The organizational change takes years. Domain teams resist new accountabilities. Executives underestimate the training and culture-building required.

Ignoring Data Product Quality

Domains declare data products “done” when the pipeline runs. A data product is done when consumers can use it reliably. Freshness monitoring, completeness checks, and SLA dashboards are not optional.

The Platform Team Trap

Platform teams frequently revert to being a centralized bottleneck by handling too many implementation details for domain teams. The platform team’s job is to make self-service so easy that domains never need to file a ticket. If the platform team is executing domain-specific work, the self-serve principle has failed.

Self-Serve Platform Design

Measure the platform by one metric: time-to-first-data-product for a new domain team, with zero platform team tickets. Build Terraform module templates, CI scaffolding, and dbt project generators so a domain team can provision GCS buckets, IAM bindings, schema registration, and a DataHub entry by running a single make new-data-product command. Any work that a domain team asks the platform team to do manually is a product backlog item for the platform — automate it or document the self-serve path.

Data Mesh and Medallion Architecture: How They Coexist

Medallion architecture (bronze/silver/gold) is not incompatible with data mesh — it operates at a different level of abstraction.

Within a single data product, the domain team may implement a medallion pattern:

Payments Domain — internal implementation
─────────────────────────────────────────────────────────────────
Source API  ──►  payments_bronze (raw)  ──►  payments_silver (clean)
                                         ──►  payments_transactions_v2 (data product, public interface)

The data product exposes only the gold/silver layer — the bronze layer is the domain’s internal implementation detail. Consumers see payments_transactions_v2, not the internal pipeline stages.

Across the mesh, different domain teams’ data products may be the inputs to other domains’ pipelines:

Domain Products (mesh nodes)          Downstream Domain Products
─────────────────────────────────────────────────────────────────
payments_transactions_v2  ─────►
orders_completed_v1       ─────►  finance_revenue_recognition_v1
customer_profiles_v3      ─────►

The dbt transformation layer within a domain’s data product pipeline implements the silver-to-gold transforms that make data consumer-ready.

Data Mesh Architecture References

Graph View

Backlinks