03 - Docker Compose

20 min read

Docker Compose


flowchart LR
  A[Windows host<br/>C:\\Users\\aperi\\DEV\\ESG] --> B[docker-compose.yml]
  B --> C[stoxx-db]
  B --> D[stoxx-pipeline]
  B --> E[stoxx-dashboard]
  F[stoxx-airflow VM<br/>bq-wh-nb] --> G[systemd<br/>stoxx-airflow.service]
  G --> H[/home/alexper_recovery_gmail_com/app/docker-compose.yaml]
  H --> I[airflow-apiserver]
  H --> J[airflow-scheduler]
  H --> K[airflow-worker]
  H --> L[airflow-dag-processor]
  H --> M[airflow-triggerer]
  H --> N[postgres + redis]
  J --> O[Cloud Run jobs<br/>stage-fetch, bronze-load,<br/>transforms, serving]
  K --> O

Local And Remote Topology

The same Docker Compose command family is driving two very different environments. The local stack is a developer-facing integration environment. The VM stack is a production-like orchestration environment that must survive reboots and start automatically.

Stack Topology Resolution

This section uses real resolved config and live object state rather than generic YAML fragments. That matters because Compose is doing interpolation, path resolution, project naming, and mount construction that are not obvious from a quick skim of the files.

AspectLocal Windows stackLive Airflow VM stackOperational consequence
Compose fileC:\Users\aperi\DEV\ESG\docker-compose.yml/home/alexper_recovery_gmail_com/app/docker-compose.yamlNever assume one file describes both environments.
Compose project namestoxxappContainer, volume, and network names differ completely.
Startup ownerInteractive operator on Windowsstoxx-airflow.service under systemdVM boot recovery is automated; local startup is manual.
Stateful serviceSQL Server 2022PostgreSQL 16 and Redis 7.2Database engine, volume paths, and health semantics differ.
Port exposureHost 1434 and 8080 locallyHost 8080 on the VM API server onlyInternal VM services are not published to the host.
Host-mounted pathsWindows repo paths/home/alexper_recovery_gmail_com/app/{dags,logs,config,plugins}Filesystem behavior and line-ending risk differ.

PowerShell | docker compose config | resolve the local compose services and named volumes

Before docker compose up, before cleanup, or when you need to know which objects the local stack is supposed to create. It is typically triggered by the compose file has changed or the local object names are unclear. PowerShell on the Windows host in C:\Users\aperi\DEV\ESG. These are read-only config resolution commands. Confirm the service keys and named volumes that the local compose file actually defines.

Resolves the local service keys from the compose file.

docker compose -f C:\Users\aperi\DEV\ESG\docker-compose.yml config --services
db
dashboard
pipeline

Resolves the named volumes declared by the same compose file.

docker compose -f C:\Users\aperi\DEV\ESG\docker-compose.yml config --volumes
sqlserver_data
pipeline_logs

These outputs prove that the local stack is not an Airflow stack. It is a three-service development stack with one database service and two application services, plus two named volumes. The live local container inventory in 01-container-lifecycle only showed stoxx-db running, but the compose model itself still defines all three services and both named volumes.

PowerShell | docker inspect | inspect the local SQL Server mount model

When persistence, bootstrap scripts, or schema seed files are not behaving as expected on the local stack. It is typically triggered by SQL Server starts but does not see bootstrap content, or data persistence is unclear. PowerShell on the Windows host. This is a read-only inspect command against the existing stoxx-db container. Show exactly which host files and which named volume the local SQL Server container is using.

Inspects the concrete mount set of the running local SQL Server container.

docker inspect stoxx-db --format '{{json .Mounts}}'
[{"Type":"volume","Name":"stoxx_sqlserver_data","Source":"/var/lib/docker/volumes/stoxx_sqlserver_data/_data","Destination":"/var/opt/mssql","Driver":"local","Mode":"rw","RW":true,"Propagation":""},{"Type":"bind","Source":"C:\\Users\\aperi\\DEV\\ESG\\db\\ddl\\bronze_schema.sql","Destination":"/docker-entrypoint-initdb.d/bronze_schema.sql","Mode":"rw","RW":true,"Propagation":"rprivate"},{"Type":"bind","Source":"C:\\Users\\aperi\\DEV\\ESG\\db\\seed\\countries.sql","Destination":"/docker-entrypoint-initdb.d/countries.sql","Mode":"rw","RW":true,"Propagation":"rprivate"},{"Type":"bind","Source":"C:\\Users\\aperi\\DEV\\ESG\\docker\\db-init.sh","Destination":"/docker-entrypoint-initdb.d/db-init.sh","Mode":"rw","RW":true,"Propagation":"rprivate"},{"Type":"bind","Source":"C:\\Users\\aperi\\DEV\\ESG\\db\\ddl\\gold_schema.sql","Destination":"/docker-entrypoint-initdb.d/gold_schema.sql","Mode":"rw","RW":true,"Propagation":"rprivate"},{"Type":"bind","Source":"C:\\Users\\aperi\\DEV\\ESG\\db\\ddl\\silver_schema.sql","Destination":"/docker-entrypoint-initdb.d/silver_schema.sql","Mode":"rw","RW":true,"Propagation":"rprivate"},{"Type":"bind","Source":"C:\\Users\\aperi\\DEV\\ESG\\docker\\db-entrypoint.sh","Destination":"/entrypoint.sh","Mode":"rw","RW":true,"Propagation":"rprivate"}]

This is the real local persistence model. SQL Server data itself lives in the named volume stoxx_sqlserver_data, while the bootstrap scripts and seed DDL are bind-mounted directly from the repo. That means data survives container recreation, but bootstrap script edits are instantly visible because they come straight from the host filesystem.

FlagSyntaxDescription
config --servicesdocker compose config --servicesLists only the service keys after interpolation and merge.
config --volumesdocker compose config --volumesLists the named volumes declared by the compose file.
-fdocker compose -f <file> ...Forces the intended compose file.
--formatdocker inspect --format '{{json .Mounts}}'Extracts only the mount information instead of the full inspect document.

Live Airflow VM Compose Stack

The Airflow VM is a different Compose estate entirely. It has its own compose file, its own .env, its own bind-mount root, and a systemd wrapper that starts Compose on boot. This is the operational truth of the current VM on April 13, 2026.

Live Compose Host Inspection

This section matters because the repo still contains older Airflow documentation for a docker run + startup-script deployment. The live VM no longer matches that model. The only safe way to operate the host is to inspect the running compose project and its boot unit directly.

Linux | compose.yaml / .env | read the live compose file and its redacted environment keys

Before modifying the VM stack, before restarting the Airflow services, or when a DAG or job binding seems to come from the wrong environment. It is typically triggered by the Airflow VM behavior no longer matches the repo docs or expected service names. Linux shell on stoxx-airflow. This is a read-only inspection of the live compose file and .env. Show the actual service graph, healthchecks, bind mounts, and environment keys used by the live VM.

The live compose file anchors a shared x-airflow-common block, uses CeleryExecutor, and drives seven long-running services plus one one-shot init service. The VM .env below is shown with secrets redacted but with live operational values preserved.

Shows the shared x-airflow-common definition, CeleryExecutor wiring, bind-mounted project paths, and the live .env values that bind the VM stack to the bq-wh-nb Cloud Run jobs.

x-airflow-common:
  &airflow-common
  image: ${AIRFLOW_IMAGE_NAME:-stoxx-airflow:3.2.0}
  build: .
  env_file:
    - .env
  environment:
    &airflow-common-env
    AIRFLOW__CORE__EXECUTOR: CeleryExecutor
    AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
    AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql+psycopg2://airflow:airflow@postgres/airflow
    AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
    AIRFLOW__CORE__EXECUTION_API_SERVER_URL: "http://airflow-apiserver:8080/execution/"
  volumes:
    - ${AIRFLOW_PROJ_DIR:-.}/dags:/opt/airflow/dags
    - ${AIRFLOW_PROJ_DIR:-.}/logs:/opt/airflow/logs
    - ${AIRFLOW_PROJ_DIR:-.}/config:/opt/airflow/config
    - ${AIRFLOW_PROJ_DIR:-.}/plugins:/opt/airflow/plugins
  user: "${AIRFLOW_UID:-50000}:0"
AIRFLOW_UID=50000
AIRFLOW_IMAGE_NAME=stoxx-airflow:3.2.0
AIRFLOW_PROJ_DIR=/home/alexper_recovery_gmail_com/app
_AIRFLOW_WWW_USER_USERNAME=admin
_AIRFLOW_WWW_USER_PASSWORD=<redacted>
FERNET_KEY=<redacted>
AIRFLOW__API_AUTH__JWT_SECRET=<redacted>
AIRFLOW__API_AUTH__JWT_ISSUER=airflow
GCP_PROJECT_ID=bq-wh-nb
STAGE_BUCKET=stoxx-stage-bucket
GCP_REGION=europe-west1
STAGE_FETCH_JOB=stoxx-stage-fetch
STAGE_LOAD_JOB=stoxx-bronze-load
TRANSFORM_JOB=stoxx-transforms
SERVING_JOB=stoxx-serving

This is a compose-based Airflow 3.2.0 stack with explicit job bindings to the current bq-wh-nb Cloud Run jobs. It also shows the real bind-mount root on the VM: /home/alexper_recovery_gmail_com/app, not /home/airflow.

Linux | systemd | read the unit that starts Compose on boot

After reboot problems, after Compose file changes, or when you need to know how the VM converges back to the desired stack. It is typically triggered by the VM came back but the stack composition looks wrong or stale containers remain after service edits. Linux shell on stoxx-airflow. This is a read-only host configuration inspection. Show the real boot-time mechanism that starts the Airflow compose project.

Shows the live systemd unit that starts the Airflow compose stack on the VM.

[Unit]
Description=STOXX Airflow Docker Compose Stack
Requires=docker.service
After=docker.service network-online.target
Wants=network-online.target
 
[Service]
Type=oneshot
RemainAfterExit=yes
User=alexper_recovery_gmail_com
Group=docker
WorkingDirectory=/home/alexper_recovery_gmail_com/app
Environment=HOME=/home/alexper_recovery_gmail_com
ExecStart=/usr/bin/docker compose up -d --remove-orphans
ExecStop=/usr/bin/docker compose down
TimeoutStartSec=0
 
[Install]
WantedBy=multi-user.target

This unit is the reason the live VM must be treated as a systemd-managed Compose host, not as a metadata-startup-script host. It uses docker compose up -d --remove-orphans from the real working directory and keeps the stack converged to the current compose model on boot.

Linux | journalctl | read the actual boot trail from systemd

After reboot, after enabling a new service, or when airflow-init and health-gated dependencies do not start in the expected order. It is typically triggered by the VM finished booting but the Airflow stack looks incomplete or misordered. Linux shell on stoxx-airflow. This is a read-only journal query against the compose wrapper service. Prove the real boot sequence of the current VM stack.

Reads the current-boot journal for the systemd unit that starts Compose on the VM.

gcloud compute ssh stoxx-airflow --project bq-wh-nb --zone europe-west1-b --tunnel-through-iap --command "sudo journalctl -u stoxx-airflow.service -b --no-pager"
Apr 13 14:30:23 stoxx-airflow systemd[1]: Starting STOXX Airflow Docker Compose Stack...
Apr 13 14:30:23 stoxx-airflow docker[26536]:  Container app-postgres-1 Running
Apr 13 14:30:23 stoxx-airflow docker[26536]:  Container app-redis-1 Running
Apr 13 14:30:24 stoxx-airflow docker[26536]:  Container app-airflow-init-1 Starting
Apr 13 14:31:09 stoxx-airflow docker[26536]:  Container app-airflow-init-1 Exited
Apr 13 14:31:09 stoxx-airflow docker[26536]:  Container app-airflow-apiserver-1 Healthy
Apr 13 14:31:09 stoxx-airflow systemd[1]: Finished STOXX Airflow Docker Compose Stack.

This is the clean boot path of the current VM. PostgreSQL and Redis are treated as prerequisites, airflow-init is expected to run and exit, and the systemd unit only finishes after the long-running stack has been brought up under Compose.

FlagSyntaxDescription
--remove-orphansdocker compose up -d --remove-orphansRemoves containers from older compose models that are no longer declared.
-ujournalctl -u stoxx-airflow.serviceFilters the journal to the specific systemd unit.
-bjournalctl -bLimits the log view to the current boot only.
--no-pagerjournalctl --no-pagerPrevents interactive paging, which is required for scripted capture.
--projectgcloud compute ssh ... --project bq-wh-nbForces the command to the live GCP project.
--zonegcloud compute ssh ... --zone europe-west1-bTargets the correct zone.
--tunnel-through-iapgcloud compute ssh ... --tunnel-through-iapUses IAP for VM access.
--commandgcloud compute ssh ... --command "<cmd>"Runs a non-interactive remote command.

Real Problems And Setup Trail

Compose is where environment drift and startup problems become visible first. This section records the problems that were actually encountered while updating this chapter. Some apply to the live Airflow 3.2.0 compose stack. Others belong to the older startup-script-based Airflow generation preserved in the repo and are included because they explain why the repo docs no longer match the running VM.

Environment Drift Reconciliation

The first compose problem encountered during this chapter rewrite was not a YAML syntax error. It was documentation drift. The repo and the live VM no longer described the same Airflow deployment.

Linux | docker inspect | reconcile the repo-era Airflow description with the live VM

Before changing the Airflow VM, before following older repo docs, or before documenting the runtime. It is typically triggered by the repo says docker run and stoxx-index-intelligence, but the live host looks different. Live VM inspection via gcloud compute ssh and Docker container labels. Read-only. Identify the actual compose project path, compose file, and project name that the live VM is using. Problem: The repo still documents an older Airflow deployment pattern, while the live VM on April 13, 2026 is a Compose stack in a different GCP project. The first live queries against stoxx-index-intelligence failed, and the repo’s earlier docker run-based Airflow guide did not match the actual running containers.

Reads the live Compose labels from the running Airflow API server container.

gcloud compute ssh stoxx-airflow --project bq-wh-nb --zone europe-west1-b --tunnel-through-iap --command "docker inspect app-airflow-apiserver-1 --format '{{json .Config.Labels}}'"
{"com.docker.compose.project":"app","com.docker.compose.project.config_files":"/home/alexper_recovery_gmail_com/app/docker-compose.yaml","com.docker.compose.project.working_dir":"/home/alexper_recovery_gmail_com/app","com.docker.compose.service":"airflow-apiserver","com.docker.compose.version":"5.1.2","org.apache.airflow.version":"3.2.0"}

Diagnosis: The live host is not using the repo’s older infra/scripts/airflow-startup.sh model. It is running a Compose project named app from /home/alexper_recovery_gmail_com/app/docker-compose.yaml in project bq-wh-nb. Resolution: This chapter was rewritten against the live bq-wh-nb stack and the actual Compose labels instead of repeating the stale repo path. Validation: The label output above aligns with the live docker compose ps, .env, systemd unit, and Cloud Run job bindings captured elsewhere in this chapter. Prevention rule: Always verify the live project ID, compose working directory, and compose labels before treating existing docs as authoritative.

Healthcheck Timeout Investigation

The second compose problem is current and live. Two Airflow containers are running, producing logs, and still being marked unhealthy by Compose.

Linux | docker inspect / docker exec | capture the false-unhealthy state and time the real probe

When docker compose ps reports unhealthy but the container logs still show forward progress. It is typically triggered by airflow-dag-processor or airflow-triggerer appear degraded even though the stack is otherwise functioning. Linux shell on stoxx-airflow. Read-only inspection plus a manual execution of the same healthcheck command. Distinguish a dead process from a slow probe. Problem: On April 13, 2026, app-airflow-dag-processor-1 and app-airflow-triggerer-1 were running but marked unhealthy. The live compose file gives both services a timeout: 10s CLI-based healthcheck.

Reads Docker’s recorded health state for the live airflow-dag-processor container.

gcloud compute ssh stoxx-airflow --project bq-wh-nb --zone europe-west1-b --tunnel-through-iap --command "docker inspect app-airflow-dag-processor-1 --format '{{json .State.Health}}'"
{"Status":"unhealthy","FailingStreak":7,"Log":[{"Start":"2026-04-13T17:33:59.268912979Z","End":"2026-04-13T17:34:09.386842741Z","ExitCode":-1,"Output":"Health check exceeded timeout (10s): ... Found one alive job.\n"}]}

Runs the same Airflow job check manually and measures how long it really takes.

gcloud compute ssh stoxx-airflow --project bq-wh-nb --zone europe-west1-b --tunnel-through-iap --command "/usr/bin/time -f %E docker exec app-airflow-dag-processor-1 airflow jobs check --job-type DagProcessorJob --hostname 33e75f7b8725 && /usr/bin/time -f %E docker exec app-airflow-triggerer-1 airflow jobs check --job-type TriggererJob --hostname 04af152e5d81"
Found one alive job.
0:07.99
Found one alive job.
0:07.95

Diagnosis: The process heartbeat exists, but the CLI-based probe is running close enough to the 10s budget that Docker occasionally records it as a timeout. This is a probe-budget problem, not evidence that the Airflow job itself is dead. Resolution: The next safe infrastructure change is to increase these healthcheck timeouts above 10s or replace the CLI probe with a lighter check. That change was not applied during this documentation pass because it would mutate the live orchestration host. Validation: Manual execution returned Found one alive job. for both services in 0:07.99 and 0:07.95, proving that the job heartbeats exist even when Compose reports unhealthy. Prevention rule: If a CLI-based Airflow healthcheck routinely consumes more than about 80% of its timeout budget on the live VM, raise the timeout before the next rollout.

Live Verified Workflow

The unhealthy state and the manual probe timings above are live captures from the current VM.

Important Conceptual Note Not Executed Here

The remediation change itself was not executed here because it would require editing the live compose file and restarting active Airflow services on the VM.

Historical Setup Remediation

The repo still contains an older Airflow-on-COS deployment model. That model is no longer the live VM, but the setup failures recorded there remain important because they explain why the repo added line-ending normalization and permission-fix guidance.

Linux | sed / terraform | fix CRLF line endings in the earlier startup-script deployment

Only when working with the older metadata-startup-script deployment preserved in the repo. It is typically triggered by the VM boot log shows the shell cannot execute the startup script even though the script is present. Historical setup trail from the repo’s earlier Airflow deployment model. The commands below are the actual fix path recorded during that deployment. Keep the earlier setup history because it explains a real class of Windows-to-Linux drift. Problem: The startup script failed with env: 'bash\r': No such file or directory. The repo originally pushed infra/scripts/airflow-startup.sh into VM metadata for a docker run-based Airflow stack.

Normalizes Windows CRLF line endings in infra/scripts/airflow-startup.sh with sed, then reapplies only the Airflow VM Terraform target so the startup metadata is rewritten with Linux-safe line endings.

sed -i 's/\r$//' infra/scripts/airflow-startup.sh
terraform -chdir=infra apply -target=google_compute_instance.airflow
env: 'bash\r': No such file or directory

Diagnosis: The script had Windows CRLF line endings, so the Linux shebang and shell parser could not execute it correctly on the VM. Resolution: Normalize line endings before applying Terraform, and use the replace(file(...), "\r\n", "\n") pattern in Terraform so the metadata copy is always LF-normalized. Validation: The repo’s later infra/compute.tf changed the startup-script metadata assignment to replace(file("${path.module}/scripts/airflow-startup.sh"), "\r\n", "\n"), which is the durable infrastructure fix. Prevention rule: Any shell script that crosses from Windows editing to Linux execution should be normalized to LF before it becomes a Docker entrypoint or VM startup script.

Linux | chown / ls | fix host-path ownership in the earlier Airflow bind mounts

When an Airflow container can see mounted DAG or log paths but cannot write to them. It is typically triggered by airflow containers start, but logs fail to write or DAG files do not load because ownership is wrong. Historical setup trail from the earlier Airflow deployment model in the repo. Preserve the real mount-permission fix because the same UID pattern still exists in the current Compose-based VM stack. Problem: Host-mounted Airflow paths failed with permission errors because they were owned by the wrong UID or GID. The older Airflow stack mounted /home/airflow/dags and /home/airflow/logs into containers that ran as UID 50000, while PostgreSQL required UID 999 on its data path.

Reassigns the Airflow DAG and log directories to UID 50000, reassigns PostgreSQL data to UID and GID 999, and verifies the corrected ownership numerically with ls -ln.

sudo chown -R 50000:0 /home/airflow/dags /home/airflow/logs
sudo chown -R 999:999 /home/airflow/pgdata
ls -ln /home/airflow/
drwxr-xr-x  50000 0     dags
drwxr-xr-x  50000 0     logs
drwx------  999   999   pgdata

Diagnosis: The container runtime user IDs and the host directory ownership did not match. Airflow could not write to dags and logs, and PostgreSQL could not own pgdata safely. Resolution: Align the host directory ownership to the runtime UIDs before or immediately after the stack starts. Validation: The expected ownership pattern above is the operational proof that the host paths match the container UIDs. Prevention rule: Every bind-mounted state path should be checked against the service’s runtime UID before blaming the application itself.

Graph View

Backlinks