05 - Data Contracts

11 min read

Data Contracts

Quote

“A data contract is the API of data — it sets expectations between producers and consumers so that changes don’t break downstream systems silently.”

— Andrew Jones (data contracts advocate)

What a Data Contract Contains

ComponentDefinitionExample
SchemaColumn names, types, nullability, constraintsinstrument_isin CHAR(12) NOT NULL
SLAFreshness, availability, quality thresholds”Available by 18:00 UTC, 99.9% uptime”
SemanticsBusiness meaning of each field”close_price is the official exchange closing price in local currency”
OwnershipWho produces, who maintains, who to contact”Market Data Team owns, Index Ops consumes”
VersioningHow changes are communicated and rolled out”Semver: breaking = major, additive = minor”

The contract concept parallels API contracts in REST design — both define a stable interface between producer and consumer, with versioning and backward-compatibility guarantees.

Contract Without Enforcement

A contract YAML file that nobody validates in CI provides a false sense of safety. The contract will drift from reality within weeks: a column gets renamed, a type changes, an SLA shortens. Downstream consumers still break on schema changes — but now they’re SURPRISED because the contract said it wouldn’t happen. Enforce contracts automatically: Pydantic at ingestion, dbt tests at transform, CI checks at deployment. A contract that isn’t tested is a lie.

Enforce Contracts Automatically

Wire the contract into the pipeline at three points: (1) Pydantic model validation at ingestion time to reject non-conforming rows before they enter bronze; (2) dbt contract tests (not_null, unique, accepted_values) that run on every dbt build; (3) a CI step in GitHub Actions that runs python scripts/validate_contracts.py contracts/ on every push to contracts or models. A contract enforced at all three points cannot silently drift.

Contract-First Development Workflow

graph LR
    CONTRACT[Define Contract YAML] --> VALIDATE[Validate in CI]
    VALIDATE --> PRODUCER[Producer implements]
    VALIDATE --> CONSUMER[Consumer codes against contract]
    PRODUCER --> TEST[Contract tests in pipeline]
    TEST --> PUBLISH[Publish data]
  1. Producer and consumer agree on the contract (schema + SLA) before any code is written
  2. Contract is version-controlled alongside the pipeline code
  3. CI validates that produced data matches the contract
  4. Breaking changes require a new major version and migration period

Schema Definition Formats

FormatStrengthsWhen to Use
JSON SchemaHuman-readable, widely supportedREST APIs, config validation
Protocol BuffersStrongly typed, backward-compatible by designgRPC services, high-throughput
AvroSchema evolution built-in, compact binaryKafka/Pub/Sub messages
dbt YAMLNative to dbt, enforced at build timeWarehouse transforms
SQL DDLUniversal, everyone reads SQLDatabase tables

Data Contracts in Practice — Exported from Code

The contract examples above use YAML — a design-time format. The functional pipeline takes a different approach: contracts are GENERATED from code at runtime. Pydantic models define the schema, ColumnContext registries define the semantics, and export_contracts() serializes both into a JSON Schema file — a machine-readable contract that any consumer (including AI agents) can parse.

See functional-pipeline-architecture for the architecture and 25_py_functional_pipeline for the implementation.

Example Contract: ESG Score Feed

Contract Header and Schema

The contract header identifies the dataset, owner, and version. The schema section defines every column with type, nullability, and business description.

# contracts/esg-scores-v2.yaml — header and metadata
contract:
  name: esg-scores
  version: 2.0.0
  owner: esg-data-team
  description: "Normalized ESG scores, 0-100 scale (higher = better)"
# contracts/esg-scores-v2.yaml — schema definition
schema:
  columns:
    - name: instrument_isin
      type: string
      length: 12
      nullable: false
      description: "ISO 6166 ISIN identifier"
    - name: vendor_code
      type: string
      nullable: false
      enum: [MSCI, SUSTAINALYTICS, ISS, BLOOMBERG, CDP]
    - name: normalized_score
      type: decimal
      precision: 5
      scale: 2
      nullable: true
      constraints: { min: 0, max: 100 }
    - name: score_date
      type: date
      nullable: false
    - name: loaded_at
      type: timestamp
      nullable: false
  primary_key: [instrument_isin, vendor_code, score_date]

SLA and Change Classification

The SLA section defines freshness, availability, and quality thresholds. Breaking vs non-breaking changes follow semver: breaking = major version bump with migration period.

# contracts/esg-scores-v2.yaml — SLA and change rules
sla:
  freshness: "Updated weekly by Monday 08:00 UTC"
  availability: "99.9%"
  quality:
    completeness: ">= 95% of index universe covered"
    null_rate: "< 2% for normalized_score"
 
breaking_changes:
  - Removing a column
  - Changing a column type
  - Changing primary key
  - Narrowing an enum
 
non_breaking_changes:
  - Adding a new column (nullable)
  - Widening an enum (adding new vendor)
  - Relaxing a constraint

Example Contract: Index Constituent Feed

Constituent Schema with SCD2-Style Dates

effective_date / expiry_date pattern enables point-in-time queries. weight_pct must sum to 1.0 per index per date — a circuit-breaker quality check.

# contracts/index-constituents-v1.yaml — schema
contract:
  name: index-constituents
  version: 1.0.0
  owner: index-operations-team
 
schema:
  columns:
    - { name: index_code, type: string, nullable: false }
    - { name: instrument_isin, type: string, length: 12, nullable: false }
    - { name: effective_date, type: date, nullable: false }
    - { name: expiry_date, type: date, nullable: false, default: "9999-12-31" }
    - { name: weight_pct, type: decimal, nullable: false, constraints: { min: 0, max: 1 } }
    - { name: change_reason, type: string, nullable: true,
        enum: [REBALANCE, IPO_ADD, MERGER_REMOVE, DELIST, SPIN_OFF_ADD] }
  primary_key: [index_code, instrument_isin, effective_date]
# contracts/index-constituents-v1.yaml — SLA
sla:
  freshness: "Updated at quarterly rebalancing and on corporate actions"
  quality:
    weight_sum: "SUM(weight_pct) = 1.00000000 per index_code + date"
    completeness: "Exactly N constituents where N = target count"

Contract Testing in CI

Automated Contract Enforcement

Validates contract YAML syntax and runs dbt contract tests on every push that touches contracts or models. See github-actions-patterns for reusable workflow patterns.

# .github/workflows/contract-test.yml
name: Data Contract Validation
on:
  push:
    paths: ['contracts/**', 'models/**']
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install jsonschema pyyaml
      - run: python scripts/validate_contracts.py contracts/
      - run: dbt build --select tag:contract_test --target ci

Breaking vs Non-Breaking Contract Changes

ChangeBreaking?Action Required
Add nullable columnNoMinor version bump
Remove columnYesMajor version, migration period
Change column typeYesMajor version
Rename columnYesMajor version
Add enum valueNoMinor version bump
Remove enum valueYesMajor version
Tighten constraintYesMajor version
Relax constraintNoMinor version bump
Change SLADependsCommunicate to all consumers

Additive Changes Are Not Always Safe

“Adding a nullable column is non-breaking” is true for schema-aware consumers. But a consumer that does SELECT * and feeds the result to a fixed-width parser, a strict Avro schema, or a Pydantic model with model_config = ConfigDict(extra='forbid') will BREAK on the new column. Additive changes are safe only when ALL consumers handle unknown fields gracefully. In practice, announce additive changes and give consumers a release window, even if they’re “non-breaking.”

Safe Pattern for Additive Changes

Before adding any column, audit downstream consumers for SELECT * usage, strict Avro/Protobuf schemas, and Pydantic models with extra='forbid'. Publish the upcoming change in the contract YAML as a minor version bump with at least a two-sprint notice. Consumers should adopt model_config = ConfigDict(extra='ignore') for producer-owned schemas, and use explicit column lists (SELECT col1, col2) rather than SELECT * so new columns are invisible until they opt in.

Producer and Consumer Responsibilities

ResponsibilityProducerConsumer
Schema definitionDefines and maintainsValidates inputs against
SLA complianceMonitors and guaranteesMonitors and alerts on breach
Breaking changesPublishes new major versionMigrates within deprecation window
Quality checksValidates before publishingValidates after receiving
DocumentationMaintains contract YAMLReferences contract in their code
IncidentsNotifies consumers of issuesReports anomalies to producer

Data Contract Anti-Patterns

Anti-PatternProblemBetter Approach
No contract existsSchema changes break consumers silentlyDefine contracts before building
Contract not enforcedContract exists but nobody checksAutomate validation in CI and pipeline (see data-quality-framework)
Verbal agreements”We agreed in a meeting” is not auditableVersion-controlled YAML contracts
Producer ignores consumer needsSchema designed for producer convenienceJoint schema design sessions
No deprecation periodOld version removed immediatelyMinimum 30-day deprecation window

Graph View

Backlinks