02 - GCP Cost Monitoring and Budgets

21 min read

GCP Cost Monitoring and Budgets

Why This Matters

Cost monitoring fails in two common ways. The first is no control plane at all, where teams only discover spend at invoice time. The second is a fake control plane, where dashboards and notes assume exports, budgets, or anomaly feeds exist even though the project has never enabled them. bq-wh-nb is currently in the second category if you rely on the old placeholder chapter, so the safe path is to document exactly which surfaces exist and which do not.

Conceptual Model

Budgets, anomalies, reports, and forecasts answer different questions and should not be collapsed into one vague “cost alerting” concept.


flowchart LR
    U["Usage happens"] --> E["Billing and telemetry ingestion"]
    E --> B["Budget<br>plan threshold"]
    E --> A["Anomaly<br>historical deviation"]
    E --> R["Report<br>descriptive breakdown"]
    E --> F["Forecast<br>projected spend"]
    B --> N["Notification or automation"]
    A --> N
    R --> H["Human review"]
    F --> H

Archived Cost-Control State

The archived project state is incomplete but explicit:

  • Budgets cannot be listed because billingbudgets.googleapis.com is disabled on bq-wh-nb.
  • Pub/Sub has no topics and no subscriptions, so there is no archived delivery path for programmatic budget or anomaly notifications.
  • Cloud Scheduler is disabled, so there is no archived scheduled automation surface in the project.
  • Recommender is disabled, so FinOps hub and recommendation-backed optimization workflows are blocked.

PowerShell / Linux | gcloud | inspect the current cost-control surfaces

These commands establish whether the project has a real budgeting and automation plane or only the underlying billable services.

Attempt to list billing budgets

Run this before claiming budgets exist or before troubleshooting a missing budget notification. It is typically triggered by use it when building a cost control note, budget automation, or runbook for the current billing account. This is a read-only command against the billing account, but it still depends on the Cloud Billing Budget API being enabled for the consumer project. Verify whether budgets are queryable from the archived project context.

FieldSource columnUnit / typeMeaning
[]CLI resultJSON arrayThe command returns no budget objects before failing.
serviceError metadataSTRINGAPI service that blocked the command.
reasonError metadataSTRINGPrimary failure reason returned by Google APIs.
activationUrlError metadataSTRINGThe exact enablement URL for the missing API.

This command tries to enumerate budgets on the archived billing account.

gcloud billing budgets list --billing-account=0190CF-C61D5A-F08831 --format=json
[]
API [billingbudgets.googleapis.com] not enabled on project [bq-wh-nb].
ERROR: (gcloud.billing.budgets.list) ... Cloud Billing Budget API has not been used in project bq-wh-nb before or it is disabled.
...
service: billingbudgets.googleapis.com
reason: SERVICE_DISABLED

This is not evidence of zero budgets. It is evidence that the archived project could not query the budget surface yet.

Check whether Pub/Sub topics exist for programmatic notifications

Run this before describing budget-to-Pub/Sub or anomaly-to-Pub/Sub automation as if it already exists. It is typically triggered by use it when validating notification plumbing. This is a read-only project inventory call against Pub/Sub. Confirm whether the project currently has a message bus that could receive billing notifications.

FieldSource columnUnit / typeMeaning
JSON array lengthCLI resultINTEGERNumber of topics returned by the project inventory.

This command lists Pub/Sub topics in bq-wh-nb.

gcloud pubsub topics list --format=json
[]

No topics means there is currently no destination for budget or anomaly notifications.

Check whether Pub/Sub subscriptions exist

Run this after checking topics and before assuming any downstream consumer exists. It is typically triggered by use it when a workflow claims a Cloud Run service, function, or worker consumes billing events. This is a read-only Pub/Sub inventory lookup. Confirm whether any subscriber exists to process cost-control messages.

FieldSource columnUnit / typeMeaning
JSON array lengthCLI resultINTEGERNumber of subscriptions visible in the project.

This command lists Pub/Sub subscriptions in bq-wh-nb.

gcloud pubsub subscriptions list --format=json
[]

This confirms there is no current consumer path for cost notifications.

Check whether Cloud Scheduler exists for automation

Run this before documenting scheduled anomaly checks or budget-validation jobs. It is typically triggered by use it when a chapter or runbook mentions scheduled Slack, Cloud Run, or cleanup actions. This read-only command still depends on the Cloud Scheduler API being enabled on the project. Verify whether the project can enumerate scheduler jobs today.

FieldSource columnUnit / typeMeaning
serviceError metadataSTRINGAPI that blocked the command.
reasonError metadataSTRINGGoogle API failure reason.

This command checks for scheduler jobs in europe-west1.

gcloud scheduler jobs list --location=europe-west1 --format=json
[]
API [cloudscheduler.googleapis.com] not enabled on project [bq-wh-nb].
ERROR: (gcloud.scheduler.jobs.list) PERMISSION_DENIED: Cloud Scheduler API has not been used in project bq-wh-nb before or it is disabled.
...
service: cloudscheduler.googleapis.com
reason: SERVICE_DISABLED

Scheduled cost automation is not available until the API is enabled.

Check whether recommendations are available

Run this before describing FinOps hub or recommendation-driven optimization as an active workflow. It is typically triggered by use it when a review depends on rightsizing or idle-resource recommendations. This is a read-only Recommender API call scoped to the project and zone. Verify whether recommendation-backed optimization is queryable today.

FieldSource columnUnit / typeMeaning
serviceError metadataSTRINGAPI that blocked the lookup.
reasonError metadataSTRINGReturned failure reason.

This command attempts to list VM rightsizing recommendations for bq-wh-nb.

gcloud recommender recommendations list `
  --recommender=google.compute.instance.MachineTypeRecommender `
  --location=europe-west1-b `
  --project=bq-wh-nb `
  --format=json
[]
API [recommender.googleapis.com] not enabled on project [bq-wh-nb].
ERROR: (gcloud.recommender.recommendations.list) PERMISSION_DENIED: Recommender API has not been used in project bq-wh-nb before or it is disabled.
...
service: recommender.googleapis.com
reason: SERVICE_DISABLED

FinOps hub cannot be treated as active and queryable until the recommendation surface is enabled.

FlagSyntaxDescription
--billing-account--billing-account=0190CF-C61D5A-F08831Targets the billing account rather than the active project alone.
--location--location=europe-west1Scopes regional control-plane queries.
--recommender--recommender=google.compute.instance.MachineTypeRecommenderSelects the recommendation family being queried.
--format--format=jsonPreserves the raw API failure metadata and empty-array result.

The Cost Control Plane Is Not Wired Yet

bq-wh-nb can spend money today, but budgets, scheduled automation, and recommendation-driven optimization are not active project capabilities yet. Treat any document that assumes they already exist as inaccurate.

Build The Control Plane In This Order

  • Enable standard Cloud Billing export first so there is invoice-grade data.
  • Enable billingbudgets.googleapis.com, pubsub.googleapis.com, cloudscheduler.googleapis.com, and recommender.googleapis.com only when you are ready to use them.
  • Create a Pub/Sub topic and subscriber before you document programmatic notifications as operational reality.

PowerShell / Linux | BigQuery | analyze workload signals while billing export is absent

Without billing export, the safest temporary cost proxy is workload telemetry. BigQuery job metadata cannot replace invoice data, but it does show who is scanning data and which tables dominate analytical activity.

Query BigQuery jobs by user for the last 30 days

Run this when billing export is missing but you still need to understand which principals drive query activity. It is typically triggered by use it during spend reviews, sudden query spikes, or TCO work. This is a read-only SQL query against region-europe-west1.INFORMATION_SCHEMA.JOBS_BY_PROJECT. Rank users and service accounts by query count, bytes processed, and slot consumption.

FieldSource columnUnit / typeMeaning
user_emailJOBS_BY_PROJECT.user_emailSTRINGPrincipal that submitted the query job.
query_countCOUNT(*)INTEGERNumber of completed query jobs in the time window.
total_bytes_processedSUM(total_bytes_processed)INTEGER bytesTotal logical bytes processed by that principal.
total_slot_msSUM(total_slot_ms)INTEGER msAggregate slot time consumed by that principal.

This query summarizes BigQuery query activity by principal for the last 30 days.

SELECT
  user_email,
  COUNT(*) AS query_count,
  SUM(total_bytes_processed) AS total_bytes_processed,
  SUM(total_slot_ms) AS total_slot_ms
FROM `region-europe-west1`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
WHERE creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
  AND job_type = 'QUERY'
  AND state = 'DONE'
GROUP BY user_email
ORDER BY total_bytes_processed DESC
LIMIT 20
user_emailquery_counttotal_bytes_processedtotal_slot_ms
alexper.recovery@gmail.com572013468312531
bq-wh-sa@bq-wh-nb.iam.gserviceaccount.com8583859334117943
github-actions-sa@bq-wh-nb.iam.gserviceaccount.com12616435296

The current query volume is small in absolute terms, but the service account consumes far more slot time than the human user, which is a useful signal for scheduled or pipeline-driven work.

Query the most-referenced tables by bytes processed

Run this when you need to find which datasets or tables are most likely to drive analytical cost. It is typically triggered by use it after a spike, before designing budgets, or during TCO modeling. This is a read-only SQL query that unnests referenced_tables from JOBS_BY_PROJECT. Identify the tables and metadata surfaces most often touched by recent queries.

FieldSource columnUnit / typeMeaning
project_idreferenced_tables.project_idSTRINGProject that owns the referenced table.
dataset_idreferenced_tables.dataset_idSTRINGDataset containing the referenced table.
table_idreferenced_tables.table_idSTRINGTable or metadata view referenced by the query.
query_countCOUNT(*)INTEGERNumber of queries touching that object.
total_bytes_processedSUM(total_bytes_processed)INTEGER bytesAggregate bytes processed across those queries.

This query ranks referenced tables by bytes processed over the last 30 days.

SELECT
  referenced_tables.project_id AS project_id,
  referenced_tables.dataset_id AS dataset_id,
  referenced_tables.table_id AS table_id,
  COUNT(*) AS query_count,
  SUM(total_bytes_processed) AS total_bytes_processed
FROM `region-europe-west1`.INFORMATION_SCHEMA.JOBS_BY_PROJECT,
UNNEST(referenced_tables) AS referenced_tables
WHERE creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
  AND job_type = 'QUERY'
  AND state = 'DONE'
GROUP BY project_id, dataset_id, table_id
ORDER BY total_bytes_processed DESC
LIMIT 20
project_iddataset_idtable_idquery_counttotal_bytes_processed
bq-wh-nbstoxx_silvereurostoxx50_ohlcv3762318280
bq-wh-nbstoxx_goldTABLES1041943040
bq-wh-nbstoxx_goldCOLUMNS1041943040
bq-wh-nbstoxx_silverindex_dim226380036
bq-wh-nbstoxx_bronzetrading_calendar163270232
bq-wh-nbstoxx_goldscores_daily52269270

This result shows a useful anti-pattern: metadata introspection through INFORMATION_SCHEMA is itself a visible analytical workload. In small environments the cost is tiny, but it still proves that schema exploration can dominate bytes processed when the business tables are small.

FlagSyntaxDescription
--use_legacy_sql--use_legacy_sql=falseForces GoogleSQL so INFORMATION_SCHEMA works as written.
--format--format=prettyjsonReturns structured result rows when the query is run through bq query.

PowerShell / Linux | Logging / Monitoring APIs | inspect supporting cost signals

When billing export is absent, resource telemetry still helps isolate likely cost paths such as NAT use and logging growth.

Read recent Cloud NAT flow logs

Run this when network egress or NAT gateway usage is suspected to be part of a cost increase. It is typically triggered by use it during TCO reviews or after seeing unexplained internet-facing traffic. This is a read-only Cloud Logging query over NAT gateway flow logs. Confirm whether a VM is actively using Cloud NAT and where the traffic is going.

FieldSource columnUnit / typeMeaning
endpoint.vm_namejsonPayload.endpoint.vm_nameSTRINGVM using the NAT gateway.
gateway_namejsonPayload.gateway_identifiers.gateway_nameSTRINGNAT gateway serving the connection.
dest_ipjsonPayload.connection.dest_ipSTRINGRemote endpoint reached through NAT.
nat_ipjsonPayload.connection.nat_ipSTRINGPublic NAT IP address used for the translation.

This command reads two recent NAT flow log entries from the project.

gcloud logging read 'resource.type="nat_gateway"' --limit=2 --freshness=30d --format=json
[
  {
    "jsonPayload": {
      "connection": {
        "dest_ip": "185.125.188.57",
        "nat_ip": "34.52.196.121",
        "src_ip": "10.132.0.8"
      },
      "endpoint": {
        "vm_name": "stoxx-vm",
        "region": "europe-west1",
        "zone": "europe-west1-b"
      },
      "gateway_identifiers": {
        "gateway_name": "stoxx-nat",
        "router_name": "stoxx-router"
      }
    },
    "resource": {
      "type": "nat_gateway"
    }
  }
]

This does not quantify the bill, but it proves that stoxx-vm is actively using stoxx-nat, so NAT is not merely an unused configuration artifact.

Query Cloud Logging billing bytes via the Monitoring API

Run this when you need to know whether logging volume is drifting upward before it becomes a billed ingestion issue. It is typically triggered by use it during weekly FinOps review or after noisy service changes. This is a read-only Monitoring API call using the active gcloud access token. Retrieve recent points for logging.googleapis.com/billing/bytes_ingested.

FieldSource columnUnit / typeMeaning
resource_typemetric.labels.resource_typeSTRINGResource classification for the ingested log bytes.
endTimepoints[].interval.endTimeTIMESTAMPEnd of the sampled minute interval.
int64Valuepoints[].value.int64ValueINTEGER bytesBytes ingested for that interval.

This PowerShell call queries recent logging.googleapis.com/billing/bytes_ingested time-series data for bq-wh-nb. The excerpt below is trimmed to three recent points for readability.

$token = gcloud auth print-access-token
$headers = @{ Authorization = "Bearer $token" }
$project = "projects/bq-wh-nb"
$filter = [System.Uri]::EscapeDataString('metric.type="logging.googleapis.com/billing/bytes_ingested"')
$intervalEnd = (Get-Date).ToUniversalTime().ToString("o")
$intervalStart = (Get-Date).ToUniversalTime().AddDays(-1).ToString("o")
$url = "https://monitoring.googleapis.com/v3/$project/timeSeries?filter=$filter&interval.endTime=$intervalEnd&interval.startTime=$intervalStart&view=FULL"
Invoke-RestMethod -Headers $headers -Uri $url
{
  "metric": {
    "labels": {
      "resource_type": "audited_resource"
    },
    "type": "logging.googleapis.com/billing/bytes_ingested"
  },
  "points": [
    {
      "interval": { "endTime": "2026-04-13T15:37:00Z" },
      "value": { "int64Value": "3530" }
    },
    {
      "interval": { "endTime": "2026-04-13T15:36:00Z" },
      "value": { "int64Value": "2352" }
    },
    {
      "interval": { "endTime": "2026-04-13T15:35:00Z" },
      "value": { "int64Value": "1265" }
    }
  ]
}

Current logging ingestion is tiny. The operational value here is not the absolute number. It is that the project exposes a metric you can trend, threshold, and compare over time even before billing export is enabled.

FlagSyntaxDescription
--freshness--freshness=30dLimits the Cloud Logging query horizon.
--limit--limit=2Returns a compact result set for manual inspection.
viewview=FULLRequests full time-series points from the Monitoring API.

Budget Vs Anomaly Vs Report Vs Forecast

Each surface answers a different operational question. Mixing them together creates noisy automation and poor runbooks.

SurfaceOperational questionArchived status in removed projectCorrect action
Budget”Are we crossing a planned spend threshold?”Blocked by disabled Budget APIEnable the API, then define thresholds and destinations.
Anomaly”Is today materially different from recent history?”Native anomaly review not validated hereUse workload proxies now; enable billing export and anomaly surfaces next.
Report”What changed by service, SKU, project, or label?”Console surface exists conceptually, but export-backed detail is absentTreat reports as a post-export step, not as an archived validated workflow.
Forecast”If the month continues like this, where do we land?”Console surface exists conceptually; archived project lacked export-backed spend historyUse conservative manual forecasting until billing export is enabled.

Current product note: budgets and anomalies are not the same channel

Current Cloud Billing pricing guidance makes two practical points explicit: budgets and anomaly detection are free control-plane features, but Pub/Sub delivery for either one incurs standard Pub/Sub charges. Treat those optional notification paths as automation plumbing, not as the core budgeting feature itself.

Important Conceptual Or Console Workflows Not Safely Executed Here

Some Google Cloud cost-management features are important enough to document even when the archived project could not validate them.

SurfaceWhat the platform supportsWhy it was not executed hereSafe next step
Cloud Billing ReportsSpend trend analysis and forecasting in the consoleThe project has no billing export dataset and this note stays CLI/API-groundedUse Reports after export is enabled and compare it to BigQuery export rows.
Budget email and Pub/Sub notificationsThreshold notifications and automation hooksThe Budget API is disabled and Pub/Sub is emptyEnable Budget API, create a topic, then validate delivery with a non-destructive test budget.
Anomaly detectionSpend anomaly review and optional notificationsThe project does not yet have the surrounding budget/export plumbing documented as activeEnable export first, then review anomalies against invoice-grade data.
FinOps hubRecommendation and optimization review surfacerecommender.googleapis.com is disabled hereEnable Recommender and review outputs before writing optimization automation.

Current product note: notification delivery semantics

Programmatic Cloud Billing notifications are operational signals, not exactly-once event streams. Current documentation describes repeated notification delivery during the day, and operators should build Pub/Sub consumers to be idempotent rather than assuming one clean message per threshold crossing.

Current product note: FinOps hub inputs

FinOps hub is not just a prettier budget screen. It depends on recommendation and historical-usage surfaces, so leaving Recommender disabled blocks one of the main current optimization entry points even if basic billing visibility exists elsewhere.

Recommendations / Production Rules

  • Do not confuse “no budgets returned” with “no budgets exist” until you prove the Budget API is enabled.
  • Treat workload telemetry as a proxy, not a replacement, for billing export.
  • Use budgets for plan adherence and anomalies for historical deviation; one should not replace the other.
  • Prefer selective enforcement such as stopping non-critical resources or throttling optional workloads; disabling billing for the whole project is an outage response, not a routine budget action.
  • Never automate cost response before you know which resources are safe to stop and which are business-critical.
  • Add a billing export dataset before you invest in dashboards, because every serious trend, reconciliation, and forecast depends on it.

Troubleshooting / Incident-Response Runbooks

Missing budget visibility

  • Confirm the billing account ID with gcloud billing accounts list.
  • Run gcloud billing budgets list ....
  • If the result shows SERVICE_DISABLED, enable billingbudgets.googleapis.com in the consumer project before debugging IAM any further.

Sudden BigQuery spend suspicion without billing export

  • Run the JOBS_BY_PROJECT query by principal to isolate the actor.
  • Run the referenced-tables query to find which tables or metadata scans dominate bytes processed.
  • Do not describe the result as invoice truth; describe it as a workload proxy until export exists.

Suspected network cost drift

  • Read recent NAT flow logs.
  • Confirm whether the traffic is genuine external dependency traffic or an avoidable pattern that could use Private Google Access.
  • Only then decide whether the NAT gateway is a justified recurring driver.

Suspected logging cost drift

  • Query logging.googleapis.com/billing/bytes_ingested.
  • Compare recent point density and magnitude to the pre-change baseline.
  • If the metric grows, trace the producing resource type before changing retention or exclusions blindly.

Quick Reference

QuestionArchived answer
Can the project list billing budgets today?No; billingbudgets.googleapis.com is disabled.
Is there a Pub/Sub path for billing notifications?No; topics and subscriptions are both empty.
Is scheduled cost automation available?No; Cloud Scheduler API is disabled.
Are recommendation-backed optimizations available?No; Recommender API is disabled.
What can be used right now for cost clues?BigQuery INFORMATION_SCHEMA, NAT flow logs, and logging.googleapis.com/billing/bytes_ingested.

Graph View

Backlinks