MOC: Orchestration

Orchestration determines when pipelines run, in what order, and what happens when they fail. These notes cover Apache Airflow for DAG-centric workflows, Dagster for asset-centric orchestration and modern data-platform operations, and platform-native scheduling tools for simpler recurring jobs.

mindmap
  ((Airflow))
    (core concepts)
    (DAG patterns)
    (deployment)
    (production problems)
    (troubleshooting)

Airflow

domain-airflow

Apache Airflow for programmatic workflow orchestration — architecture, DAG authoring, production deployment, common failure modes, and troubleshooting.

mindmap
  ((Dagster))
    (core concepts)
    (assets and lineage)
    (automation)
    (integrations)
    (deployment)
    (troubleshooting)

Dagster

domain-dagster

Dagster for modern data orchestration — asset modeling, declarative automation, external compute patterns, testing, production deployment, and migration from DAG-first orchestration.

mindmap
  ((Scheduling))
    (Linux cron, systemd)
    (GCP Cloud Scheduler)
    (Windows Task Scheduler)

Scheduling

domain-scheduling

Platform-native scheduling tools for recurring jobs that do not require a full orchestration framework — Linux cron and systemd timers, GCP Cloud Scheduler and Cloud Workflows, and Windows Task Scheduler.

Orchestration Cross-References

• Data Architecture — pipeline patterns that Airflow orchestrates
• docker-compose — running Airflow locally via Docker Compose
• cloud-run-jobs-vs-services — serverless targets Airflow triggers
• datadog-airflow-observability — monitoring Airflow with Datadog
• github-actions-ci-cd — deploying DAGs automatically via CI/CD
• dbt-airflow-integration — running dbt in Airflow DAGs (BashOperator, Cosmos, Cloud Run)