Airflow is the orchestration layer for the live STOXX index pipeline running on stoxx-airflow. It does not fetch market data itself, parse JSON itself, compute silver and gold tables itself, or publish Firestore documents itself. It schedules, coordinates, retries, and records the work performed by Cloud Run jobs and the downstream systems they touch.
What This Note Covers
This note defines the minimum Airflow vocabulary required to understand the live platform safely, then maps each term to the deployed stoxx_stage_yfinance DAG and the Airflow 3.2 runtime that currently orchestrates the STOXX-only pipeline in project bq-wh-nb.
What Airflow is and is not in this platform.
Which runtime components exist on stoxx-airflow, and what each one does.
How DAGs, tasks, task instances, DAG runs, retries, timeouts, connections, and XCom appear in the real STOXX pipeline.
Why the current deployment uses CeleryExecutor, a local metadata database, Redis, and Cloud Run jobs instead of placing the data-processing logic directly inside Airflow tasks.
Glossary / Key Terms
Key Terms
Term
Definition
Why it matters here
Caveat
Airflow
A workflow orchestrator that stores run state, schedules work, and coordinates task execution.
It is the control plane for the STOXX pipeline.
It is not the compute engine that transforms market data.
DAG
A Directed Acyclic Graph that defines tasks and dependencies as code.
stoxx_stage_yfinance is the pipeline contract Airflow parses and schedules.
A DAG definition is static Python code; runtime data should not shape it at import time.
DAG run
One execution of a DAG for a specific trigger and time context.
The validated end-to-end serving run is manual__2026-04-13T17:28:30Z_serving.
A DAG run can exist even when some tasks fail or are skipped.
Task
A single node in the DAG graph.
load_bronze_into_sql and build_bigquery_marts are tasks.
A task definition is not an execution record.
Task instance
One execution record of one task inside one DAG run.
Airflow stores start time, end time, and state per task instance.
Retries create multiple attempts for the same logical task instance.
Scheduler
The Airflow component that decides what can run next.
It turns the parsed DAG and task states into queued work.
If it stalls, every DAG appears broken even when workers are healthy.
Worker
The component that executes queued tasks.
The Celery worker calls the Google provider operator that starts Cloud Run jobs.
The worker is not where the STOXX pipeline data processing actually happens.
Triggerer
The component that manages deferred asynchronous work.
The live stack runs a triggerer even though the current DAG sets deferrable=False.
Having a triggerer does not mean tasks are automatically deferrable.
Metadata database
Airflow’s shared state store.
It holds DAG metadata, task states, connections, and UI state.
If it is wrong or unavailable, the platform cannot be trusted.
Executor
The strategy Airflow uses to hand queued tasks to execution slots.
The live runtime uses CeleryExecutor.
The executor choice changes scaling and failure behavior, not DAG semantics.
Connection
A named integration object Airflow uses to resolve credentials and defaults for external systems.
google_cloud_default is required for CloudRunExecuteJobOperator.
VM metadata credentials do not remove the need for the connection record itself.
XCom
Airflow’s cross-communication channel for small metadata payloads between tasks.
The Google operator pushes execution metadata that Airflow can inspect.
XCom is not used for JSON payloads, market data, or table data in this platform.
Catchup
Airflow behavior that backfills missed schedule intervals automatically.
The live DAG disables it with catchup=False.
Enabling it accidentally can create replay storms.
Data interval
The time window a scheduled DAG run represents.
It matters when DAGs are cron-driven and partitioned by logical time.
The current DAG is schedule=None, so manual runs are the operational default.
What Airflow Is In This Platform
Airflow is the system that decides when the STOXX pipeline may advance from one stage to the next. It is the place that knows that bronze loading must finish before silver transforms start, that BigQuery marts must wait for gold tables, and that Firestore publication must not run until marts are built.
Runtime Architecture
The deployed Airflow runtime is a private Compute Engine VM called stoxx-airflow. The VM runs Airflow 3.2.0 in Docker Compose, with Postgres as the metadata database, Redis as the Celery broker, and one Celery worker that launches Google Cloud Run jobs.
flowchart LR
U[Operator]
UI[Airflow UI and API<br>airflow-apiserver]
S[Scheduler]
D[Dag Processor]
T[Triggerer]
W[Celery Worker]
PG[(Postgres<br>metadata DB)]
R[(Redis<br>broker)]
CR1[Cloud Run Job<br>stoxx-stage-fetch]
CR2[Cloud Run Job<br>stoxx-bronze-load]
CR3[Cloud Run Job<br>stoxx-transforms]
CR4[Cloud Run Job<br>stoxx-serving]
U --> UI
UI --> PG
S --> PG
D --> PG
T --> PG
S --> R
R --> W
D --> S
W --> CR1
W --> CR2
W --> CR3
W --> CR4
Airflow Does Not Process Market Data
The live DAG deliberately keeps Airflow thin. The scheduler and worker do not fetch yfinance data row by row, do not parse every JSON file, and do not execute the medallion SQL transformations inline. They invoke external jobs that own those responsibilities.
Airflow Is Not The ETL Engine
Putting the market-data parsing, SQL loading, or BigQuery mart logic directly inside long-lived Airflow Python tasks would move heavy compute into the control plane. That makes retries slower, worker saturation more likely, and failure recovery harder.
[!success] Airflow Owns Orchestration Only
The live design keeps Airflow responsible for dependency control, retries, timeouts, visibility, and manual reruns. Cloud Run jobs own the data movement and transformation logic.
The Live Control Plane
This section maps the core runtime components to the actual Airflow VM and shows the command outputs that prove the current topology.
Runtime Components On stoxx-airflow
This subsection shows which Airflow services are actually running now and how to interpret them.
Inspect The Running Airflow Services
Run this after deployment, after any Compose restart, or whenever the UI suggests a service-level problem. It is typically triggered by A DAG is missing, tasks are not advancing, or container health is in doubt. Run from a workstation with gcloud access. The command is read-only. It tunnels through IAP because the VM has no public IP. Confirm that the Airflow API server, scheduler, dag processor, triggerer, worker, Postgres, and Redis are all present and healthy.
This command SSHes through IAP to stoxx-airflow and asks Docker Compose for the live container state.
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTSapp-airflow-apiserver-1 stoxx-airflow:3.2.0 "/usr/bin/dumb-init …" airflow-apiserver 2 hours ago Up 2 hours (healthy) 0.0.0.0:8080->8080/tcp, [::]:8080->8080/tcpapp-airflow-dag-processor-1 stoxx-airflow:3.2.0 "/usr/bin/dumb-init …" airflow-dag-processor 2 hours ago Up 2 hours (healthy) 8080/tcpapp-airflow-scheduler-1 stoxx-airflow:3.2.0 "/usr/bin/dumb-init …" airflow-scheduler 2 hours ago Up 2 hours (healthy) 8080/tcpapp-airflow-triggerer-1 stoxx-airflow:3.2.0 "/usr/bin/dumb-init …" airflow-triggerer 2 hours ago Up 2 hours (healthy) 8080/tcpapp-airflow-worker-1 stoxx-airflow:3.2.0 "/usr/bin/dumb-init …" airflow-worker 2 hours ago Up 2 hours (healthy) 8080/tcpapp-postgres-1 postgres:16 "docker-entrypoint.s…" postgres 5 hours ago Up 5 hours (healthy) 5432/tcpapp-redis-1 redis:7.2-bookworm "docker-entrypoint.s…" redis 5 hours ago Up 5 hours (healthy) 6379/tcp
The important operational reading is straightforward:
airflow-apiserver serves the UI and Airflow API.
airflow-scheduler decides which task instances can queue next.
airflow-dag-processor parses DAG files into scheduler-consumable metadata.
airflow-worker executes queued operator code.
airflow-triggerer is available for deferred tasks.
postgres and redis are not optional sidecars; they are required state dependencies for CeleryExecutor.
Read The Scheduler Role From Live Logs
Run this when the scheduler might be unhealthy or after a restart when you need to see whether it actually came back. It is typically triggered by DAG runs remain queued, task instances do not advance, or the scheduler heartbeat is suspect. Run from the same VM shell path. The command is read-only and tails scheduler logs. Prove that the scheduler has loaded the executor, started its main loop, and is responding to health probes.
This tails the scheduler container log so the operator can confirm that scheduling has actually resumed.
The Loaded executor: :CeleryExecutor: line matters because it proves the runtime is not using a local single-process executor. The repeated GET /health ... 200 lines matter because the container health check is succeeding, which means the service is alive rather than merely started.
Flag
Syntax
Description
--project
gcloud compute ssh ... --project=bq-wh-nb
Selects the active GCP project that contains the VM.
--zone
gcloud compute ssh ... --zone=europe-west1-b
Selects the VM zone.
--tunnel-through-iap
gcloud compute ssh ... --tunnel-through-iap
Reaches the private VM without requiring a public IP.
Airflow’s state model is only useful if the scheduler can see the DAG, the executor can queue work correctly, and the provider operators can resolve their connections.
Verify That The DAG Is Registered
Run this after copying a new DAG file, after restarting Airflow services, or when the UI does not show the workflow. It is typically triggered by A newly deployed DAG does not appear, or a known DAG appears paused or missing. This is a read-only CLI check executed inside the Airflow worker container. Confirm that stoxx_stage_yfinance is present in the DagBag and visible to Airflow.
This command filters the Airflow DAG catalog to the live STOXX DAG.
The important field here is False in the paused column. Earlier in the rollout, the DAG was visible but still paused. Airflow can know about a DAG and still refuse to schedule it until that flag is cleared.
Verify The Executor And Google Connection
Run this on first bootstrap, after image rebuilds, or when Google operators start failing unexpectedly. It is typically triggered by tasks queue but do not launch Cloud Run jobs, or provider operators complain about missing credentials or connection IDs. These are read-only Airflow CLI calls executed inside the worker container. Prove that the runtime uses CeleryExecutor and that google_cloud_default exists in the metadata database.
The first command prints the configured executor. The second prints the stored Google connection record that the Cloud Run operator relies on.
The connection output is operationally important for one reason: the Google provider resolved its default connection name. The VM service account provides the underlying credentials through Application Default Credentials, but the Airflow connection record is still the object that the operator expects to exist.
Flag
Syntax
Description
exec -T
docker compose exec -T airflow-worker ...
Runs a command inside the worker container without allocating a pseudo-TTY, which keeps non-interactive output clean.
config get-value
airflow config get-value core executor
Reads the effective Airflow configuration value for a given section and key.
connections get
airflow connections get google_cloud_default
Prints the stored Airflow connection record.
dags list
airflow dags list
Prints DAGs visible to the current Airflow runtime.
The Core Airflow Objects In The Live DAG
The best way to learn Airflow safely is to map the vocabulary to a real DAG instead of an isolated tutorial script. The current platform uses one manually triggered DAG that fans out into silver transforms, converges into gold and serving tasks, and finishes with Firestore validation.
DAG Definition And Scheduling Semantics
This subsection shows the actual DAG declaration and explains what each top-level option means in the live deployment.
Read The Real DAG Definition
The following snippet is taken directly from stoxx_stage_yfinance.py. It is the real DAG that Airflow currently parses on stoxx-airflow.
Real DAG Declaration
with DAG( dag_id="stoxx_stage_yfinance", description="Fetch STOXX bronze-stage JSON from yfinance into GCS via Cloud Run", start_date=pendulum.datetime(2026, 4, 13, tz="Europe/Prague"), schedule=None, catchup=False, max_active_runs=1, default_args={ "retries": 1, "retry_delay": timedelta(minutes=5), "execution_timeout": timedelta(minutes=45), }, tags=["stoxx", "bronze", "gcs", "yfinance", "cloud-run"],) as dag:
Each field has a concrete operational meaning:
dag_id="stoxx_stage_yfinance" is the stable Airflow identifier used everywhere else: logs, task instances, CLI inspection, and the UI.
start_date=... Europe/Prague anchors the DAG in the business timezone used for the demo environment.
schedule=None means Airflow will not create recurring runs on its own. Runs are manual or API-triggered.
catchup=False means Airflow will not backfill historical intervals automatically.
max_active_runs=1 serializes full pipeline runs so the demo environment does not overlap bronze, silver, gold, BigQuery, and Firestore publication windows.
retries=1, retry_delay=5 minutes, and execution_timeout=45 minutes apply to all tasks by default unless a task overrides them.
schedule=None Is Intentional
This DAG is currently designed for controlled demonstration and validation runs. Turning it into a recurring cron schedule without first defining the partitioning, backfill policy, and overlap policy would create avoidable replay risk.
[!success] Manual Orchestration Keeps The Blast Radius Small
The current choice makes every full run explicit. Operators can reset the recent window, trigger the DAG once, observe the whole chain, and validate the serving state deterministically.
Tasks, Operators, And Dependencies
The DAG is a graph of CloudRunExecuteJobOperator tasks. Each task launches a purpose-built Cloud Run job and then records the execution outcome in Airflow.
Read The Real Task Graph
The following dependency block is the real orchestration skeleton from the deployed DAG.
This graph expresses five different Airflow concepts at once:
fetch_bronze_stage_into_gcs is the extract-and-land boundary.
load_bronze_into_sql is the bronze persistence boundary.
The three transform tasks are a controlled fan-out.
build_gold_scores and build_gold_index_performance are a fan-in followed by serial gold construction.
sync_gold_to_bigquery, build_bigquery_marts, publish_serving_to_firestore, and validate_serving_layer are the publication path.
The current DAG does not use Task Groups, branching, dataset scheduling, or sensors. The graph is intentionally explicit because the pipeline is linear with one parallel silver stage.
Read One Real Operator Definition
This operator definition is representative of the live pattern. The worker does not execute transformation code locally; it tells Cloud Run which job to run and what arguments to pass.
job_name=SERVING_JOB_NAME binds the task to the Cloud Run job stoxx-serving.
args=["--mode=build-marts"] tells the serving job to execute only the BigQuery mart step.
deferrable=False means the worker slot stays occupied while Airflow waits for the Cloud Run execution to finish.
Connections, Hooks, And XCom In The STOXX DAG
Airflow’s integration surfaces are present in the platform, but they are used selectively.
How google_cloud_default Is Used
The DAG does not set gcp_conn_id explicitly on each task. The Google provider falls back to google_cloud_default, which must exist in the metadata database. When the operator ran successfully, the task test log showed the provider resolving credentials through google.auth.default().
2026-04-13T15:27:22.088482Z [info] Getting connection using `google.auth.default()` since no explicit credentials are provided.
That single line explains the full credential stack:
Airflow resolves the connection object by name.
The connection contains no embedded secret.
The provider then uses the VM service account via Application Default Credentials.
Why XCom Is Present But Not A Data Bus
The Google operator still pushes metadata to XCom, but the pipeline does not move datasets through Airflow. The task test logs show the operator’s XCom push point:
bronze, silver, and gold tables live in SQL Server on stoxx-vm
replica and marts live in BigQuery datasets such as stoxx_gold and stoxx_marts
serving documents live in Firestore database main
Airflow only stores lightweight execution metadata and state transitions.
DAG Runs And Task Instances In The Live Pipeline
The most important operational Airflow concept is that a DAG definition is static code, while DAG runs and task instances are runtime records. The same DAG can have many runs. Each run contains one task instance per task, with its own state and timestamps.
Read A Successful Full DAG Run
This subsection uses the validated serving run to show exactly what a completed Airflow pipeline looks like in the metadata database.
Inspect The Task States For The Successful Serving Run
Run this after a full DAG execution, during incident review, or while proving that a rollout succeeded end to end. It is typically triggered by you need to know which tasks ran, in what order, and whether the whole graph finished successfully. This is a read-only Airflow CLI command executed inside the worker container. Print the task-instance state table for a specific DAG run and use it as the authoritative run ledger.
This command reads task-instance state for the validated end-to-end serving run.
Triggerer, Deferrable Operators, And Why They Matter Here
The live runtime includes a healthy triggerer container, so the platform is ready for deferred asynchronous patterns. The current DAG does not use them yet because every CloudRunExecuteJobOperator is declared with deferrable=False.
This matters for capacity planning:
with deferrable=False, the worker holds the task slot while Airflow waits for the Cloud Run execution to complete
with a deferrable pattern, the worker could hand off the wait state to the triggerer and free capacity for other work
The current choice is acceptable because the DAG is single-run, manual, and demonstration-oriented. If the platform becomes scheduled and runs multiple DAGs or higher parallelism, converting suitable external-wait tasks to deferrable execution is one of the first efficiency upgrades to evaluate.
When To Care About The Triggerer
Start treating the triggerer as a capacity feature rather than a background service when these conditions are true:
several long-running external jobs are active at the same time
worker slots become the bottleneck rather than Cloud Run quotas
the DAG spends more time waiting on external execution than doing operator work
What To Remember
Airflow in this environment is a stateful orchestration control plane with a simple, explicit contract:
the DAG defines the allowed order of work
the scheduler decides when tasks may run
the worker launches provider operators
the metadata database records what happened
the actual data processing happens in Cloud Run, SQL Server, BigQuery, and Firestore
That separation is the foundation for every other Airflow note in this chapter. The next note builds on it by showing which DAG patterns the live STOXX pipeline actually uses and why those patterns were chosen.