Observability Strategy Matrix
Quote
“I’ll start with observability tools only — if you know which tools look but don’t touch, you can diagnose problems without introducing new ones.”
— Brendan Gregg, Systems Performance (2013)
“Distributed systems are pathologically unpredictable. It’s impossible to predict the myriad states of partial failure various parts of the system might end up in.”
— Cindy Sridharan, Distributed Systems Observability (2018)
Summary
This note is the monitoring decision layer for the Elysium platform: it defines what each major component should emit as metrics, logs, alerts, and dashboards, and gives the investigation order, severity model, and tool-selection rules needed to turn raw observability data into consistent triage across SQL Server, Airflow, BigQuery, Cloud Run, Pub/Sub, storage, VMs, and the pipeline as a whole.
Pillars and triage model
- Maps metrics, logs, and traces to the actual data-pipeline investigation flow, with metrics as the first signal, logs as the failure explanation, and traces as the cross-service path reconstruction.
- Treats observability as a layered diagnostic system rather than a collection of unrelated dashboards.
Per-component monitoring matrix
- Defines the key metrics, logs, alert conditions, and first response patterns for SQL Server, Airflow, BigQuery, Cloud Run, Pub/Sub, GCS, Firestore, GCE VMs, and the end-to-end pipeline.
- Uses each component section to answer what to watch, why it matters, and which implementation pages hold the concrete tool configuration.
Alerting and dashboard strategy
- Establishes a severity framework, dashboard segmentation strategy, and the practical split between Datadog and GCP-native monitoring surfaces.
- Connects dashboards and alerts back to the matrix so the same components are visible, actionable, and consistently prioritized.
Operations and safety
- Covers anti-patterns such as alert noise, metrics without owners, and dashboards without response playbooks.
- Warnings: start investigations in the right order, avoid duplicating the same alert in multiple systems, and keep component ownership, thresholds, and response paths explicit.
- Strategy surfaces: the severity framework, dashboard strategy, and Datadog-vs-GCP-native section are the note’s main governance guides.
Glossary
Observability strategy matrix
A cross-component reference that maps each platform surface to the metrics, logs, alerts, and dashboards required to operate it safely.
It matters here because the note is intended to answer what should be observed before any implementation detail is chosen.
Strategy before tooling
This matrix is a design surface, not a configuration surface. It tells you which signals matter so later dashboards and monitors can be built deliberately instead of reactively.
Metric
A numeric time-series signal that captures system behavior such as latency, utilization, backlog, or freshness.
It matters here because the matrix uses metrics as the cheapest, fastest first indication that something has drifted or failed.
Detection not diagnosis
Metrics tell you that a boundary was crossed, not why. Treat them as triage entry points, then drill into logs and traces for explanation.
Log
A record of discrete events, errors, state changes, or contextual messages emitted by an application or infrastructure service.
It matters here because the matrix uses logs to explain failures after a metric or alert indicates a problem.
Highest-detail evidence
Logs are usually where the real failure message lives, but they are too verbose to be the first thing operators stare at for every component all the time.
Trace
A distributed execution record that shows how one request, run, or workflow moved across multiple services and spans.
It matters here because pipeline incidents often cross service boundaries where a single log stream or dashboard is not enough.
Use when the path matters
Traces are most valuable when latency or failure is distributed. They add cost and complexity, so use them where multi-hop causality actually needs reconstruction.
Investigation order
The recommended sequence for triage: start with metrics, then drill into logs, then use traces if the failure crosses services.
It matters here because consistent operator behavior reduces mean time to identify the right layer of failure.
Cheap to expensive workflow
This order is not arbitrary. It moves from cheap aggregated signals to verbose evidence to cross-system path analysis, which keeps diagnosis fast and disciplined.
Alert condition
A threshold, absence rule, or state change that should trigger human or automated response.
It matters here because each component section in the matrix turns raw signals into concrete action boundaries.
Thresholds need ownership
An alert without an owner or an agreed response is just noise. The matrix only becomes useful when each condition has a real response path attached.
Severity framework
The classification scheme that ranks incidents and alerts by business impact and urgency.
It matters here because the note uses severity to keep SQL outages, stale pipelines, and lower-grade monitoring drift from being treated the same way.
Prioritization protects attention
If every alert is treated as urgent, nothing is. Severity is what preserves on-call focus for the failures that actually threaten data freshness or platform safety.
Dashboard
A curated visual surface that groups the most important signals for one system, service, or operational role.
It matters here because the note translates the matrix into concrete dashboard ownership by component and by operator workflow.
Visualization with purpose
Dashboards should compress decision-relevant signals, not mirror every raw metric. If they are not helping an operator decide what to do next, they are decoration.
Response pattern
A standard first action or investigation path tied to a specific alert or degraded signal.
It matters here because the matrix does not stop at detection; it pairs each high-signal alert class with an initial operational move.
Detection must lead somewhere
A good monitoring design shortens the path from symptom to action. Response patterns are what keep dashboards and alerts from ending at awareness only.
Data freshness
The measure of how current a dataset or pipeline output is compared with its expected update cadence.
It matters here because freshness is one of the most important business-facing observability signals across the entire Elysium platform.
Silent failure signal
A pipeline can be technically healthy while still delivering stale data. Freshness monitoring closes that gap between infrastructure health and business usefulness.
Pipeline SLA
The committed completion or freshness window a data pipeline must meet to remain operationally acceptable.
It matters here because several matrix sections treat SLA misses as the key severity boundary that converts slow execution into an incident.
Business clock not system clock
SLAs are about externally meaningful timing, not just whether jobs eventually finish. Monitoring needs to encode the deadline that actually matters to consumers.
Datadog vs GCP-native split
The strategic choice of which signals should live in Datadog versus in Cloud Monitoring, Logging, Trace, and related GCP-native services.
It matters here because the note explicitly separates tool choice from signal design and uses both stacks where they are strongest.
Avoid duplicate observability sprawl
If the same alert or dashboard lives in too many systems, operators stop trusting any single source of truth. Pick the primary surface intentionally.
Anti-pattern
A recurring monitoring design mistake that adds noise, hides failures, or slows triage instead of improving it.
It matters here because the note closes with the pitfalls that most often corrupt otherwise good observability programs.
Noise is an operational failure
Too many low-value alerts, duplicated dashboards, or ownerless metrics do not just clutter the platform. They actively make real incidents harder to detect and respond to.
The Three Pillars Applied to Data Pipelines
The three pillars from observability-deep-dive > The Three Pillars (Metrics, Logs, Traces) Applied to Data Pipelines map directly to triage workflow:
- Metrics tell you something is wrong — a number crossed a threshold
- Logs tell you what is wrong — the error message, the stack trace, the failed query
- Traces tell you where in the pipeline it went wrong — which task, which service, which hop introduced the delay or failure
Investigation Order
Start every investigation with metrics (cheap, pre-aggregated), drill into logs (verbose, expensive), and use traces only when the failure spans multiple services.
Master Monitoring Matrix — Per Component
SQL Server
Dashboard: DBA Dashboard | Config: datadog-sql-server-integration > Built-in SQL Server Metrics Collected by Datadog, gcp-cloud-monitoring-deep-dive > Ops Agent: SQL Server Configuration
Key metrics:
- Buffer cache hit ratio — percentage of pages served from memory vs disk. Below 95% means the buffer pool is too small or queries are scanning too much data
- Page Life Expectancy (PLE) — seconds a page stays in the buffer pool before eviction. Below 300s indicates memory pressure
- Wait stats — top wait types (PAGEIOLATCH = disk bottleneck, LCK_M = lock contention, CXPACKET = parallelism overhead)
- Deadlocks/sec — count of deadlock victims. Any value > 0 needs investigation
- Disk % — percentage of disk capacity used. SQL Server crashes when full
- CPU % — sustained high CPU indicates query inefficiency or under-provisioned VM
- Active connections — connection pool saturation indicator
- Batch requests/sec — workload throughput baseline
Key logs:
- Error log (severity >= 16) — SQL Server errors that affect user sessions. Severity 16 = user error, 17+ = resource/system issues
- Slow queries (>5s) — queries exceeding duration threshold, captured via Extended Events or Datadog deep database monitoring
- Deadlock XML — full deadlock graph from
system_healthsession, shows which queries and resources were involved - Login failures — failed authentication attempts. Spikes may indicate brute-force attacks or misconfigured connection strings
SQL Server alerts are only useful when each distress signal has a first response
Database incidents escalate quickly because the same bottleneck can look like a query problem, a storage problem, or a concurrency problem. The monitor needs both a trigger and an immediate diagnostic path.
SQL Server distress signals
- PLE < 300s — P2: memory pressure is evicting pages faster than they’re being read. Queries will slow as disk I/O increases
- Disk > 85% — P1: SQL Server will crash if the disk fills. Transaction log growth, TempDB spills, or backup files are common causes
- Deadlocks > 0/min — P2: transactions are being killed. Pipeline MERGE operations may fail and need retry logic
- CPU > 90% sustained 5min — P2: queries are CPU-bound. Check for missing indexes, implicit conversions, or parameter sniffing
SQL Server first response
- PLE low: check buffer pool size and identify large table scans via
sys.dm_exec_query_stats; add indexes or increase VM memory- Disk > 85%: shrink or archive old backup files; verify TempDB auto-growth is not runaway; resize disk online with
resize2fsif on Linux- Deadlocks: capture the deadlock graph from
system_healthXEvent session; addWITH (NOLOCK)to read-heavy queries or reorder update sequences in MERGE logic- CPU high: run
sys.dm_exec_requeststo find the blocking query; check for implicit type conversions or missing covering indexes
Airflow
Dashboard: Pipeline Watch | Config: datadog-airflow-observability > Key Metrics Reference, datadog-dashboards > Airflow Orchestration Dashboard
Key metrics:
- DAG duration — end-to-end time per DAG run. Baseline drift indicates data volume growth or infrastructure degradation
- Task failure rate — percentage of tasks ending in FAILED state after all retries exhausted
- Scheduler heartbeat lag — seconds since the scheduler last reported alive. Gap = scheduler is down
- DAG parse time — seconds to parse all DAG files. Above 30s means too many DAGs or expensive module-level imports
- Pool utilization — percentage of pool slots in use. 100% = tasks are queuing
- Zombie tasks — tasks marked as running but with no active process. Indicates worker crashes
Key logs:
- Task stdout/stderr — output from each task’s execution. First place to look for transform errors, SQL failures, API timeouts
- Scheduler logs — DAG parsing errors, scheduling decisions, heartbeat status
- Executor logs — worker allocation, task state transitions, resource exhaustion
Airflow alerts need to distinguish orchestration failure from DAG design drift
An SLA miss, a dead scheduler, and a slow parser all stop fresh data in different ways. The callout pair should tell the operator whether to inspect a task, a service, or the DAG code itself.
Airflow orchestration failures
- SLA miss — P1: pipeline didn’t complete within its defined SLA window. Data freshness is at risk
- Task failure after retries — P2: a task exhausted all retry attempts. Manual investigation needed
- Heartbeat > 60s — P1: scheduler is down. No new tasks will be scheduled until it recovers
- Parse time > 30s — P3: slow DAG parsing delays scheduling. Usually caused by expensive imports at module level
Airflow first response
- SLA miss: check task logs for the failed or slow task; verify upstream dependencies completed; trigger a manual backfill if data is recoverable
- Task failure after retries: inspect the final task log for the root exception; fix the root cause before manually clearing the failed task instance
- Heartbeat > 60s: SSH to the Airflow VM and check
systemctl status airflow-scheduler; restart the scheduler service and monitor for recurrence- Parse time > 30s: move expensive imports inside task functions rather than at module level; reduce the number of active DAG files if the DAG folder has grown large
BigQuery
Dashboard: Cost + Performance | Config: gcp-cloud-monitoring-deep-dive > BigQuery, gcp-cloud-monitoring-deep-dive > INFORMATION_SCHEMA Queries for Job-Level Monitoring
Key metrics:
- Bytes scanned/query — direct cost driver at $6.25/TB. The single most important cost metric
- Slot utilization — percentage of available slots in use. High utilization = queries queue
- Query count — total queries per period. Baseline for anomaly detection
- Error rate — percentage of queries failing. Non-zero needs investigation
- Duration p50/p95/p99 — query latency distribution. p99 drift indicates growing tables or missing partitions
Key logs:
- Audit logs (BigQueryAuditMetadata) — every query executed, who ran it, bytes scanned, cost. The authoritative cost analysis source
- Job failures — queries that failed with errors. Check for quota exceeded, syntax errors, permission issues
BigQuery monitoring must separate cost drift from actual job failure
BigQuery can look healthy while quietly overspending, or look slow because the wrong storage design is being scanned. Each alert needs an explicit investigation path back to the offending job.
BigQuery cost and failure drift
- Bytes scanned > threshold — P3: a query is scanning more data than expected. Likely a missing partition filter or
SELECT *- Failed queries > 0 — P3: investigate immediately. Common causes: DML quota exceeded, table deleted, permission revoked
- Duration p99 > 5min — P3: slowest queries are degrading. Check for unpartitioned tables or slot starvation
BigQuery first response
- Bytes over threshold: find the offending query in
INFORMATION_SCHEMA.JOBS_BY_PROJECTordered bytotal_bytes_processed; add aWHERE DATE(_PARTITIONTIME) = ...filter or enablerequire_partition_filteron the table- Failed queries: check the
error_resultfield inINFORMATION_SCHEMA.JOBS; for quota errors, reduce scheduling concurrency; for permission errors, audit IAM role assignments- p99 duration high: identify unpartitioned tables being full-scanned; add partition and cluster keys, or purchase additional slot capacity if slot starvation is the cause
Cloud Run
Dashboard: Service Health | Config: gcp-cloud-monitoring-deep-dive > Built-in Metrics for Cloud Run Services
Key metrics:
- Request count — total requests per period. Baseline for capacity planning
- Latency p50/p95/p99 — response time distribution. p99 captures cold-start impact
- Error rate (5xx) — percentage of server errors. Any sustained rate > 1% is a problem
- Cold starts — count of instances starting from zero. High rate = min-instances too low
- Instance count — active container instances. Correlate with request count for efficiency
- Memory % — container memory utilization. Above 90% risks OOM kills
Key logs:
- Stdout/stderr from container — application logs. First place to look for errors
- Crash logs — container exit with non-zero code. OOM, unhandled exceptions, timeout
Cloud Run alerts need to distinguish bad revisions from scaling pressure
The same symptom bucket can come from a crashing build, a cold-start problem, or an undersized container. The operator needs the likely failure mode and the first recovery move together.
Cloud Run service degradation
- Error rate > 1% — P2: sustained server errors affecting users or downstream consumers
- Latency p99 > 10s — P3: slowest requests are unacceptably slow. Check for cold starts or upstream dependency issues
- Memory > 90% — P2: close to OOM kill. Increase memory allocation or fix memory leaks
- OOM kills > 0 — P1: container is being killed for exceeding memory. Data loss possible if writes are in progress
Cloud Run first response
- Error rate > 1%: check container stdout/stderr logs via
gcloud logging read; look for unhandled exceptions or upstream dependency timeouts; redeploy previous revision if a recent deploy is the cause- Latency p99 high: check cold start frequency and set
--min-instances=1for latency-sensitive services; profile the slowest requests with Cloud Trace- Memory > 90%: increase
--memoryon the service revision; profile heap allocation if a memory leak is suspected- OOM kills: set
--min-instances=1to reduce churn; increase--memory; verify the pipeline checkpoints progress so a restart does not reprocess already-written data
Pub/Sub
Dashboard: Messaging Health | Config: gcp-cloud-monitoring-deep-dive > Pub/Sub
Key metrics:
- Unacked message count (backlog) — messages delivered but not acknowledged. Growing backlog = consumer can’t keep up
- Oldest unacked age — age of the oldest unacknowledged message. Shows how far behind the consumer is
- Publish/pull latency — time to publish or pull a message. Spikes indicate Pub/Sub service issues
- Dead letter count — messages moved to DLQ after max delivery attempts. Non-zero = systematic processing failure
Key logs:
- DLQ messages — messages that failed processing repeatedly. Contains the original payload and error context
- Subscription errors — delivery failures, acknowledgement timeouts, permission issues
Pub/Sub monitoring has to separate lag from irreversible message loss
A growing backlog can often be recovered with scale, but an aging subscription or DLQ spike can turn into lost processing if the response is slow.
Pub/Sub backlog and loss risk
- Backlog > threshold — P2: consumer is falling behind. Common causes: consumer crash, slow processing, insufficient concurrency
- Oldest unacked > 5min — P2: messages are aging. If retention window is short, data loss is imminent
- DLQ > 0 — P2: messages are failing permanently. Investigate the DLQ for error patterns
Pub/Sub first response
- Backlog growing: check the consumer Cloud Run service for crash loops; increase
--max-instancesand--concurrencyto scale out processing; verify acknowledgement deadlines are long enough for slow messages- Oldest unacked > 5min: increase the subscription’s message retention duration immediately to prevent expiry; scale up consumers in parallel; if messages have expired, replay from the source if available
- DLQ > 0: pull a sample from the DLQ to read the original payload and error; fix the consumer bug; republish DLQ messages after the fix is deployed
Cloud Storage (GCS)
Dashboard: Storage Dashboard | Config: gcp-cloud-monitoring-deep-dive > Cloud Storage
Key metrics:
- Object count — total objects per bucket. Sudden drops indicate accidental deletion
- Total bytes — storage volume per bucket. Tracks growth for capacity and cost planning
- Request count by type — Class A (writes) vs Class B (reads). Unusual write spikes may indicate runaway pipeline
Key logs:
- Access logs — who accessed which objects. Useful for audit and debugging access issues
- Lifecycle actions — objects transitioned to Nearline/Coldline/Archive or deleted by lifecycle rules
Storage monitoring must tell accidental deletion apart from growth drift
Buckets usually fail quietly: a lifecycle rule, a runaway writer, or an operator mistake all show up as object-count movement before users notice.
GCS deletion and growth anomalies
- Unexpected deletes — P2: object count dropped without a known lifecycle rule or pipeline action. Possible accidental deletion
- Object count anomaly — P3: sudden spike or drop outside normal growth patterns
GCS first response
- Unexpected deletes: check Cloud Audit Logs for
storage.objects.deleteevents in the affected bucket; if accidental, restore from a versioned bucket (enable object versioning on all landing-zone buckets); identify the IAM principal that issued the delete and restrict permissions if necessary- Object count anomaly: correlate with pipeline run history; a spike may indicate a runaway write loop — check Cloud Run job execution counts and add a maximum-objects-per-run guard to the pipeline
Firestore
Dashboard: Real-Time Store | Config: gcp-cloud-monitoring-deep-dive > Firestore
Key metrics:
- Read/write ops/sec — operation throughput. Baseline for cost projection (0.18/100K writes)
- Active connections — concurrent client connections. Spikes indicate consumer issues
- Document count — total documents. Unexpected growth may indicate a write loop
Key logs:
- Security rule denials — requests blocked by Firestore security rules. May indicate misconfigured rules or unauthorized access
- Quota warnings — approaching operation or storage quotas
Firestore alerts need to separate write contention from access-policy problems
Hot documents and denied clients both surface as degraded operations, but they require completely different fixes.
Firestore hotspotting and denials
- Write rate > 80% quota — P2: approaching the per-document write limit (1 write/sec). Contention will cause latency spikes
- Security denials > 0 — P3: investigate whether rules are too restrictive or an unauthorized client is probing
Firestore first response
- Write rate near quota: batch writes using
WriteBatchinstead of individual document writes; distribute heartbeat state across multiple documents keyed by pipeline name to avoid hotspotting on a single document- Security denials: check Cloud Audit Log for the denied request’s principal and resource path; update security rules to permit legitimate access or revoke the client’s credentials if unauthorized
GCE VMs
Dashboard: VM Health | Config: datadog-agent-sql-vm, gcp-cloud-monitoring-deep-dive > Compute Engine VMs
Key metrics:
- CPU % — processor utilization. Sustained > 90% indicates under-provisioned VM or runaway process
- Memory % — RAM utilization. High memory + swap activity = performance degradation
- Disk % — disk capacity used. SQL Server and Airflow VMs are the highest risk
- IOPS — disk operations per second. High IOPS + high latency = disk bottleneck
- Network I/O — bytes in/out. Spikes during pipeline runs are normal; sustained spikes are not
Key logs:
- Syslog — OS-level events: service starts/stops, kernel warnings, SSH logins
- OOM kills — kernel killed a process for exceeding available memory. Check
dmesgfor the victim - systemd failures — services that failed to start or crashed. SQL Server and Airflow run as systemd services
VM alerts need to separate host saturation from outright service loss
A busy VM can often limp along, but a full disk, OOM kill, or dead systemd unit turns into an outage. The monitoring pair should show both the symptom and the first stabilizing action.
VM saturation and service loss
- CPU > 90% sustained 10min — P2: VM is compute-bound. Right-size or optimize the workload
- Disk > 85% — P1: disk filling up. SQL Server transaction logs, TempDB, or backup files are common culprits
- OOM detected — P1: a process was killed for memory. Data corruption possible if SQL Server was the victim
- Service stopped — P1: SQL Server or Airflow systemd service is down. Pipeline is blocked
VM first response
- CPU > 90%: identify the top process with
toporhtop; for SQL Server, runsys.dm_exec_requeststo find the high-CPU query; right-size the VM machine type if the workload is consistently high- Disk > 85%: run
df -hto identify the full partition; archive or delete old backup files; resize the disk withgcloud compute disks resizefollowed byresize2fs- OOM: check
dmesg | grep -i "killed process"for the victim; increase VM memory or fix the memory-leaking process; verify SQL Server data consistency after an OOM kill- Service stopped: run
systemctl restart airflow-schedulerorsystemctl restart mssql-server; checkjournalctl -u <service>for the failure reason before restarting
The Data Pipeline
Dashboard: Data Quality | Config: gcp-pipeline-health-and-sla > Data Freshness Monitoring, datadog-custom-queries
Key metrics:
- Data freshness — time since the last successful pipeline run updated the target tables. The primary SLA metric
- Row count/run — rows processed per pipeline execution. Baseline for anomaly detection (±20% = investigate)
- Schema drift — columns added, removed, or type-changed since last run. Catches upstream API changes
- Duplicate rate — percentage of duplicate rows in target tables. Non-zero after dedup = logic bug
Key logs:
- Transform logs — output from each pipeline stage. Includes row counts, timing, validation results
- Validation gate results — PASS/FAIL for each quality check. The first place to look when data quality degrades
Pipeline monitors have to catch freshness, volume, and integrity drift separately
A stale table, a row-count spike, and duplicate records all mean the pipeline is unhealthy, but they point to different stages in the data flow.
Pipeline freshness and integrity drift
- Stale data > SLA — P1: pipeline hasn’t updated within the defined freshness window. Dashboard is showing old data
- Row count anomaly (>20%) — P2: sudden increase (bad join explosion) or decrease (filter bug, missing source data)
- Schema change detected — P3: upstream schema changed. Contract tests should have caught this — investigate why they didn’t
- Duplicates > 0 — P2: deduplication logic failed. MERGE key mismatch or race condition in concurrent loads
Pipeline first response
- Stale data: check Cloud Run job execution history for the pipeline; if no execution exists, verify the Cloud Scheduler trigger fired; re-trigger manually with
gcloud run jobs executeafter identifying the root cause- Row count anomaly: compare today’s count against the 7-day average in BigQuery
__TABLES__or thepipeline_freshnesstable; for spikes, check for duplicate loads; for drops, check for missing source partitions- Schema change: run
check_schema_drift()against the current table and the reference schema JSON; if a column was renamed upstream, update the reference schema and pipeline mapping; add a contract test to catch this earlier- Duplicates: identify the duplicate key values with the duplicate detection query; re-run the MERGE with the correct natural key; verify the dedup logic covers all load concurrency scenarios
Alert Severity Framework
Severity Determines Response
Every alert in datadog-alerting > Deadlock Alert Monitor and gcp-cloud-monitoring-deep-dive > Alerting Policies must map to exactly one of these levels.
| Severity | Response Time | Notification Channel | Example Conditions |
|---|---|---|---|
| P1 — Critical | 15 min, 24/7 page | PagerDuty phone call + Slack #incidents | Pipeline down, data loss risk, SLA breach imminent, deadlocks blocking production |
| P2 — High | 30 min during business hours | PagerDuty alert + Slack #alerts | Task failure after retries, CPU > 90% sustained, disk > 85%, DLQ messages detected |
| P3 — Medium | 4 hours | Slack #alerts | Latency degradation (p99 drift), row count anomaly (>20%), slot utilization high |
| P4 — Low | Next business day | Slack #monitoring-info | Schema drift detected, cost anomaly, cold start frequency increase, login failures |
Severity only works when paging urgency is rationed deliberately
If every monitor pages like an outage, operators learn to distrust the signal and real incidents lose attention.
Alert fatigue from over-paging
Avoid making everything P1. If on-call gets paged for P3 issues, alert fatigue sets in and real P1s get ignored. Review severity assignments quarterly.
Enforce the severity framework in your alerting tool
Assign PagerDuty phone-call urgency only to P1 monitors. Map P2 to PagerDuty alert (no phone call outside business hours). Map P3 and P4 to Slack only. Schedule a quarterly severity review meeting: pull alert history, count false-positive P1 pages, and downgrade any alert that has never required immediate action. This keeps the on-call rotation sustainable and P1 responses fast.
Dashboard Strategy
Five Platform Dashboards
Each has a clear audience and refresh cadence. Implementation details live in datadog-dashboards > Pipeline Watch Dashboard, datadog-dashboards > SQL Server DBA Dashboard, and gcp-cloud-monitoring-deep-dive > Dashboards.
| Dashboard | Audience | Key Panels | Refresh | Tool |
|---|---|---|---|---|
| Pipeline Watch | Data engineers, on-call | DAG status, task durations, SLA tracker, freshness gauges | 1 min | Datadog |
| DBA Dashboard | Data engineers, DBAs | PLE, buffer cache, waits, deadlocks, query duration, disk/CPU | 1 min | Datadog |
| Cost Dashboard | Engineering leads | BigQuery bytes scanned, slot-hours, Cloud Run invocations, GCS storage growth | 1 hour | GCP Console |
| Data Quality | Data engineers, analysts | Freshness SLA, row counts, duplicate rate, schema change log | 5 min | Datadog |
| Service Health | Platform team, on-call | Cloud Run latency/errors, Pub/Sub backlog, Firestore ops, GCE VM health | 1 min | GCP Console |
Dashboards Link to Runbooks
Every dashboard should have a “last updated” indicator and a link to the relevant runbook. If a panel turns red, the viewer should know exactly which runbook to open without searching.
Datadog vs GCP-Native — When to Use Which
Datadog vs GCP Decision
Full feature comparison lives at gcp-cloud-monitoring-deep-dive > Cloud Monitoring vs Datadog — Comprehensive Comparison Table. This table gives the short decision framework.
| Scenario | Use Datadog | Use GCP-Native | Rationale |
|---|---|---|---|
| SQL Server on GCE | Yes | Agent for collection only | Datadog has richer SQL Server dashboards and query-level metrics |
| Airflow DAG monitoring | Yes | No | Datadog Airflow integration provides DAG/task-level metrics out of the box |
| BigQuery cost tracking | No | Yes | INFORMATION_SCHEMA is free and gives job-level detail Datadog cannot match |
| Cloud Run / Pub/Sub / GCS | Optional | Yes | Built-in GCP metrics are zero-config and have no per-host cost |
| Cross-service alerting | Yes | Backup only | Datadog composite monitors can correlate metrics across SQL Server, Airflow, and GCP services in a single alert |
| Audit and compliance logs | No | Yes | Cloud Audit Logs are the system of record; Datadog should not be the only copy |
Redundant telemetry is useful, but redundant paging is not
Collecting the same metric in two systems improves resilience. Letting both systems page on the same threshold creates noise and ownership confusion.
Single alert source per metric
Running both tools on the same metric is acceptable for redundancy, but alerting should fire from one tool only. Duplicate alerts from Datadog and GCP for the same condition cause confusion and double-paging.
Designate a primary alerting tool per metric category
Use the decision table above to assign alerting ownership: Datadog owns SQL Server and Airflow alerts; GCP Cloud Monitoring owns Cloud Run, Pub/Sub, GCS, and Firestore alerts. Disable the secondary tool’s alerting policies (keep the metric collection active for redundancy). Document the ownership in each alert’s description field so on-call engineers know where to look.
Anti-Patterns
Observability posture erodes through governance gaps long before tooling fails
Most monitoring programs drift because ownership, thresholds, and response discipline decay gradually. Quarterly review is the safe pattern that keeps those silent failures visible.
Observability anti-patterns
These patterns silently degrade your observability posture. Audit for them quarterly.
- Metric without an alert — collecting data nobody looks at wastes money and creates false confidence
- Alert without a runbook — an alert that fires with no documented response is just noise that trains the team to ignore pages
- Dashboard without an owner — unowned dashboards rot; panels break, thresholds drift, and nobody notices
- Logs without structure — free-text logs that require regex to parse are unsearchable at scale; always use structured JSON logging
- Threshold from a guess — alert thresholds must come from baseline data, not intuition; run a component for two weeks before setting static thresholds
- Monitoring only the happy path — if you only track success counts, a silent failure (zero rows, no errors) will never fire an alert; always monitor for the absence of expected events
- Single-tool dependency — if Datadog goes down and you have no GCP-native alerts, you are blind; critical P1 alerts should exist in both systems
Quarterly observability posture checklist
Run a 30-minute audit against this list each quarter: (1) list all custom metrics and confirm each has a linked alert; (2) list all alerts and confirm each has a runbook URL in the description; (3) list all dashboards and confirm each has a named owner and a “last verified” date; (4) review a sample of pipeline logs and confirm JSON structure is consistent; (5) review alert history and recalibrate thresholds that fired zero or more than 10 times in the quarter; (6) verify at least one P1 alert exists in both Datadog and GCP-native for each critical component.