03 - Secrets Management

21 min read

Secrets Management

Why this matters for data engineering

Secret Manager solves a very specific problem: how to let workloads fetch sensitive values at runtime without hardcoding them into source code, environment files, container images, or CI secrets. It is the right home for:

  • database passwords
  • API tokens
  • vendor credentials
  • migration-era service-account keys that have not been eliminated yet

It is not the right home for:

  • large binary artifacts
  • non-sensitive configuration that belongs in code or environment-specific config files
  • authorization policy itself, which belongs in IAM

Conceptual Model

flowchart TD
    A["Secret container<br/>metadata, IAM, aliases, rotation"] --> B["Version 1<br/>old payload"]
    A --> C["Version 2<br/>current payload"]
    A --> D["Version aliases<br/>current, previous"]
    A --> E["Secret-scope IAM"]
    A --> F["Replication and CMEK"]
    F --> G["Cloud KMS key"]
    E --> H["Workload service account"]
    H --> I["Access Secret Version"]

The control model is simple but strict:

  • IAM decides which principal can access the secret.
  • the secret container decides which version alias points where.
  • KMS decides whether Secret Manager can encrypt and decrypt the payload.

Secret Inventory and Current State

The first step is to inspect what exists already. In this project the default global endpoint only shows the automatically replicated secret; the regional secret requires a regional endpoint override.

PowerShell / Linux | gcloud secrets | inspect the current secret inventory

This subsection establishes the current global-secret state and the CMEK relationship used by the automatic secret.

List the global secret inventory

At the start of secret review, rotation planning, or incident response. It is typically triggered by you need to see what secret containers are visible on the default Secret Manager endpoint. Read-only Secret Manager inventory lookup. Establish the current set of globally addressed secrets before reviewing versions or IAM.

gcloud secrets list \
  --project=bq-wh-nb \
  --format='table(name,labels)'
NAME                    LABELS
codex-api-token-260413  {'env': 'lab', 'owner': 'codex'}

This does not mean the project only has one secret. It means the default endpoint shows the automatically replicated secret. Regional secrets are a separate operational surface.

Confirm the Secret Manager service agent used for CMEK

Before wiring Secret Manager to a Cloud KMS key. It is typically triggered by you need to know which Google-managed principal must be granted KMS rights. Service Identity API call. Surface the exact Secret Manager service agent email for the project.

gcloud beta services identity create \
  --service=secretmanager.googleapis.com \
  --project=bq-wh-nb
Service identity created: service-348557092514@gcp-sa-secretmanager.iam.gserviceaccount.com

Even when the service agent already exists, this command is useful because it surfaces the exact principal that needs access to the KMS key.

Inspect the CMEK-backed automatic secret

Before rotation, alias changes, or secret-scope IAM changes. It is typically triggered by you need to understand the secret’s metadata, CMEK key, rotation schedule, and aliases. Read-only secret metadata lookup. Show the full control plane attached to the secret container.

gcloud secrets describe \
  codex-api-token-260413 \
  --project=bq-wh-nb \
  --format=json
{
  "createTime": "2026-04-13T13:59:08.599424Z",
  "etag": "\"164f5800a354b6\"",
  "labels": {
    "env": "lab",
    "owner": "codex"
  },
  "name": "projects/348557092514/secrets/codex-api-token-260413",
  "replication": {
    "automatic": {
      "customerManagedEncryption": {
        "kmsKeyName": "projects/bq-wh-nb/locations/global/keyRings/codex-sec-lab-global/cryptoKeys/secret-cmek-auto"
      }
    }
  },
  "rotation": {
    "nextRotationTime": "2026-05-01T00:00:00Z",
    "rotationPeriod": "2592000s"
  },
  "topics": [
    {
      "name": "projects/bq-wh-nb/topics/codex-sec-rotation-260413"
    }
  ],
  "versionAliases": {
    "current": "2"
  },
  "versionDestroyTtl": "86400s"
}

This one object already tells you most of the operational story:

  • automatic replication is enabled
  • a customer-managed KMS key protects the payload
  • rotation metadata exists
  • Pub/Sub notifications exist
  • alias current points to version 2
  • destroyed versions wait one day before permanent destruction

Confirm that the Secret Manager service agent can use the KMS key

Immediately after enabling CMEK or when a secret create/access call fails around encryption. It is typically triggered by you need to verify the KMS side of the dependency. Read-only IAM policy lookup on the crypto key. Prove that Secret Manager’s service agent has encrypt/decrypt access to the key.

gcloud kms keys get-iam-policy \
  secret-cmek-auto \
  --project=bq-wh-nb \
  --location=global \
  --keyring=codex-sec-lab-global \
  --format=json
{
  "bindings": [
    {
      "members": [
        "serviceAccount:service-348557092514@gcp-sa-secretmanager.iam.gserviceaccount.com"
      ],
      "role": "roles/cloudkms.cryptoKeyEncrypterDecrypter"
    }
  ],
  "etag": "BwZPV-JXyvU=",
  "version": 1
}

If this binding is missing, CMEK-backed secret operations fail even when secret IAM is correct.

Current product note: CMEK dependency chain

Secret-level IAM is only one side of a CMEK-protected design. Secret Manager also depends on its Google-managed service agent being able to use the selected KMS key, so a secret can fail for encryption reasons even when the caller has the right Secret Manager role.

FlagSyntaxDescription
--project--project=bq-wh-nbProject that owns the secret or KMS resource.
--format--format=jsonPreserves nested fields such as replication, rotation, topics, and aliases.
--location--location=globalKMS location for the automatic-replication key.
--keyring--keyring=codex-sec-lab-globalKMS key ring that owns the crypto key.

Regional Secrets and the Endpoint Override

Regional Secret Manager uses a regional endpoint. The important operational detail is that the CLI does not automatically switch endpoints just because you passed --location=europe-west1.

PowerShell / Linux | gcloud secrets and config | work with a regional secret safely

This subsection shows the exact failure mode without the regional endpoint override, then the successful regional workflow with the override in place.

Try to describe the regional secret without the regional endpoint override

Only as a diagnosis step when a regional secret command unexpectedly fails. It is typically triggered by you used --location but the command still returned an argument-format error. Read-only regional secret lookup against the default endpoint. Show the exact error that indicates the endpoint override is missing.

gcloud secrets describe \
  codex-sql-pass-ew1-260413 \
  --project=bq-wh-nb \
  --location=europe-west1 \
  --format=json
ERROR: (gcloud.secrets.describe) INVALID_ARGUMENT: The provided Secret ID [projects/bq-wh-nb/locations/europe-west1/secrets/codex-sql-pass-ew1-260413] does not match the expected format [projects/*/secrets/*]

This is not an IAM denial. It is a control-plane routing problem.

Regional secret gotcha

--location=europe-west1 is not enough by itself. The gcloud CLI still talks to the default Secret Manager endpoint unless you override api_endpoint_overrides/secretmanager.

Current product note: regional endpoint expectation

Current Secret Manager regional documentation still treats the regional endpoint as a required part of CLI and client behavior. The practical meaning is that residency is not only a storage choice; it also changes which control-plane endpoint your tools must call.

Point the CLI at the regional Secret Manager endpoint

Before regional list, describe, versions list, and similar operational commands. It is typically triggered by you need to operate on regional secrets from the CLI. Local gcloud configuration change. Route Secret Manager CLI calls to the correct regional endpoint.

gcloud config set \
  api_endpoint_overrides/secretmanager \
  https://secretmanager.europe-west1.rep.googleapis.com/
Updated property [api_endpoint_overrides/secretmanager].

List and describe the regional secret

After the regional endpoint override is in place. It is typically triggered by you need to confirm regional secret metadata or inspect version state. Read-only Secret Manager calls against the regional endpoint. Prove that the secret exists and is region-bound to europe-west1.

gcloud secrets list \
  --project=bq-wh-nb \
  --location=europe-west1 \
  --format=json
[
  {
    "annotations": {
      "owner": "codex",
      "system": "warehouse"
    },
    "createTime": "2026-04-13T14:00:09.113860Z",
    "etag": "\"164f57e68ec344\"",
    "labels": {
      "env": "lab",
      "tier": "regional"
    },
    "name": "projects/348557092514/locations/europe-west1/secrets/codex-sql-pass-ew1-260413",
    "versionDestroyTtl": "86400s"
  }
]
gcloud secrets describe \
  codex-sql-pass-ew1-260413 \
  --project=bq-wh-nb \
  --location=europe-west1 \
  --format=json
{
  "annotations": {
    "owner": "codex",
    "system": "warehouse"
  },
  "createTime": "2026-04-13T14:00:09.113860Z",
  "etag": "\"164f57e68ec344\"",
  "labels": {
    "env": "lab",
    "tier": "regional"
  },
  "name": "projects/348557092514/locations/europe-west1/secrets/codex-sql-pass-ew1-260413",
  "versionDestroyTtl": "86400s"
}

The resource name itself proves the regional boundary: locations/europe-west1/secrets/....

Return the CLI to the default Secret Manager endpoint

Immediately after a regional secret operation. It is typically triggered by you are done with the regional commands and do not want to surprise later global commands. Local gcloud configuration change. Prevent the workstation from accidentally staying pinned to a regional endpoint.

gcloud config unset api_endpoint_overrides/secretmanager
Unset property [api_endpoint_overrides/secretmanager].

Safe regional workflow

Set the regional endpoint override, run the regional commands you actually need, then unset the override immediately.

Versions, Aliases, and Rotation State

The core operational problem in Secret Manager is not storing one value. It is evolving the value safely over time while keeping consumers pointed at the correct version.

PowerShell / Linux | gcloud secrets versions | inspect alias and version state

This subsection shows how alias changes interact with disabled versions and scheduled destruction.

List the current versions of the automatic secret

Before any alias move, disable, destroy, or incident-response change. It is typically triggered by you need to know the current version states. Read-only version inventory lookup. Show which versions are enabled, disabled, and scheduled for destruction.

gcloud secrets versions list \
  codex-api-token-260413 \
  --project=bq-wh-nb \
  --format=json
[
  {
    "clientSpecifiedPayloadChecksum": true,
    "createTime": "2026-04-13T14:00:39.677562Z",
    "etag": "\"164f57e860e67a\"",
    "name": "projects/348557092514/secrets/codex-api-token-260413/versions/2",
    "replicationStatus": {
      "automatic": {
        "customerManagedEncryption": {
          "kmsKeyVersionName": "projects/bq-wh-nb/locations/global/keyRings/codex-sec-lab-global/cryptoKeys/secret-cmek-auto/cryptoKeyVersions/1"
        }
      }
    },
    "state": "ENABLED"
  },
  {
    "clientSpecifiedPayloadChecksum": true,
    "createTime": "2026-04-13T13:59:10.968064Z",
    "etag": "\"164f5800b7a43c\"",
    "name": "projects/348557092514/secrets/codex-api-token-260413/versions/1",
    "replicationStatus": {
      "automatic": {
        "customerManagedEncryption": {
          "kmsKeyVersionName": "projects/bq-wh-nb/locations/global/keyRings/codex-sec-lab-global/cryptoKeys/secret-cmek-auto/cryptoKeyVersions/1"
        }
      }
    },
    "scheduledDestroyTime": "2026-04-14T14:07:27.940037675Z",
    "state": "DISABLED"
  }
]

This is the healthy rotation picture:

  • version 2 is enabled and active
  • version 1 is disabled
  • version 1 still has a recovery window before permanent destruction

Recreate a stale alias and prove the access failure

During alias troubleshooting or runbook validation. It is typically triggered by A consumer still references an alias that points to an old disabled version. Safe metadata change followed by a read attempt. Show what a broken alias looks like in practice.

gcloud secrets update \
  codex-api-token-260413 \
  --project=bq-wh-nb \
  --update-version-aliases='current=2,previous=1'
Updated secret [codex-api-token-260413].
gcloud secrets versions access previous \
  --secret=codex-api-token-260413 \
  --project=bq-wh-nb
ERROR: (gcloud.secrets.versions.access) FAILED_PRECONDITION: Secret Version [projects/348557092514/secrets/codex-api-token-260413/versions/1] is in DISABLED state.

This is the exact stale-alias failure mode. The alias exists, but it points to a version that is no longer readable.

Remove the stale alias and confirm the steady-state metadata

Immediately after confirming the stale alias problem. It is typically triggered by the old alias should no longer be used by any consumer. State-changing secret metadata update, followed by read-only inspection. Return the secret to a clean state where only current remains.

gcloud secrets update \
  codex-api-token-260413 \
  --project=bq-wh-nb \
  --remove-version-aliases='previous'
Updated secret [codex-api-token-260413].
gcloud secrets describe \
  codex-api-token-260413 \
  --project=bq-wh-nb \
  --format=json
{
  "createTime": "2026-04-13T13:59:08.599424Z",
  "etag": "\"164f5800a354b6\"",
  "labels": {
    "env": "lab",
    "owner": "codex"
  },
  "name": "projects/348557092514/secrets/codex-api-token-260413",
  "replication": {
    "automatic": {
      "customerManagedEncryption": {
        "kmsKeyName": "projects/bq-wh-nb/locations/global/keyRings/codex-sec-lab-global/cryptoKeys/secret-cmek-auto"
      }
    }
  },
  "rotation": {
    "nextRotationTime": "2026-05-01T00:00:00Z",
    "rotationPeriod": "2592000s"
  },
  "topics": [
    {
      "name": "projects/bq-wh-nb/topics/codex-sec-rotation-260413"
    }
  ],
  "versionAliases": {
    "current": "2"
  },
  "versionDestroyTtl": "86400s"
}

The container is back in its clean steady state with one active alias.

Current product note: aliases, ETags, and version targeting

Current Secret Manager guidance pairs alias-based rotation with optimistic concurrency through etag checks. It also recommends referencing a concrete version number when the reader must be deterministic, instead of treating latest as a stable contract.

FlagSyntaxDescription
--update-version-aliases--update-version-aliases='current=2,previous=1'Creates or moves aliases to specific versions.
--remove-version-aliases--remove-version-aliases='previous'Deletes alias names that should no longer resolve.
--secret--secret=codex-api-token-260413Selects the secret whose version will be accessed.

Secret-Scope IAM and Impersonated Access

The safest access model is usually:

  • project-level metadata visibility only where needed
  • secret-level accessor on the specific secret container

PowerShell / Linux | gcloud secrets and policy-intelligence | validate least-privilege secret access

This subsection proves that the disposable lab service account can read the payload of one secret because of a secret-scope IAM binding, not because of broad project access.

Inspect the secret-level IAM policy

Before granting or troubleshooting secret access. It is typically triggered by A workload should read one secret and nothing else. Read-only IAM policy lookup on the secret container. Confirm the binding exists at secret scope.

gcloud secrets get-iam-policy \
  codex-api-token-260413 \
  --project=bq-wh-nb \
  --format=json
{
  "bindings": [
    {
      "members": [
        "serviceAccount:codex-sec-lab-260413@bq-wh-nb.iam.gserviceaccount.com"
      ],
      "role": "roles/secretmanager.secretAccessor"
    }
  ],
  "etag": "BwZPV-pDSR4=",
  "version": 1
}

This is the least-privilege pattern. The lab principal can access this secret because the binding is attached here, not to the entire project.

List visible secrets as the impersonated lab service account

After granting access and before trusting that the workload can enumerate the intended container. It is typically triggered by you need to validate effective access as the workload identity itself. Secret Manager call executed through service-account impersonation. Confirm the principal can see the target secret.

gcloud secrets list \
  --project=bq-wh-nb \
  --limit=5 \
  --format='table(name)' \
  --impersonate-service-account=codex-sec-lab-260413@bq-wh-nb.iam.gserviceaccount.com
NAME
codex-api-token-260413
WARNING: This command is using service account impersonation. All API calls will be executed as [codex-sec-lab-260413@bq-wh-nb.iam.gserviceaccount.com].

Read the current secret value as the impersonated lab service account

After the metadata listing succeeds and you need to prove payload access. It is typically triggered by you want a direct end-to-end proof that the binding actually permits version reads. Secret Manager access call executed through service-account impersonation. Validate the exact runtime path a workload would use.

gcloud secrets versions access current \
  --secret=codex-api-token-260413 \
  --project=bq-wh-nb \
  --impersonate-service-account=codex-sec-lab-260413@bq-wh-nb.iam.gserviceaccount.com
ghp_codex_v2_20260413
WARNING: This command is using service account impersonation. All API calls will be executed as [codex-sec-lab-260413@bq-wh-nb.iam.gserviceaccount.com].

The value was returned through a short-lived token, not a JSON key file.

Troubleshoot the permission against the secret resource

When the caller and the binding look correct but the request still fails. It is typically triggered by you need IAM’s current answer for secretmanager.versions.access. Read-only Policy Troubleshooter call. Prove whether the secret access path is currently granted or denied.

gcloud policy-intelligence troubleshoot-policy iam \
  //secretmanager.googleapis.com/projects/348557092514/secrets/codex-api-token-260413 \
  --principal-email=codex-sec-lab-260413@bq-wh-nb.iam.gserviceaccount.com \
  --permission=secretmanager.versions.access \
  --format='yaml(overallAccessState,allowPolicyExplanation.allowAccessState,denyPolicyExplanation.denyAccessState)'
allowPolicyExplanation:
  allowAccessState: ALLOW_ACCESS_STATE_GRANTED
denyPolicyExplanation:
  denyAccessState: DENY_ACCESS_STATE_NOT_DENIED
overallAccessState: CAN_ACCESS

This is the cleanest proof that access is working for the intended reason.

FlagSyntaxDescription
--impersonate-service-account--impersonate-service-account=codex-sec-lab-260413@bq-wh-nb.iam.gserviceaccount.comRuns the command with a short-lived token for the workload identity.
--limit--limit=5Keeps list output small during validation.
--permission--permission=secretmanager.versions.accessPermission whose effective access you want Troubleshooter to evaluate.

Production Rules

  • Grant roles/secretmanager.secretAccessor on the secret container, not at project scope, unless the workload truly must read many secrets.
  • Use aliases like current to decouple consumers from raw version numbers, but remove stale aliases immediately after rotation.
  • Treat regional secrets as a first-class design choice, not an afterthought. Residency requirements affect both API endpoints and runbooks.
  • Use CMEK only when the key lifecycle and KMS access model are understood operationally. CMEK without service-agent KMS access is just a broken deployment.
  • Prefer runtime fetches through metadata-backed credentials, WIF, or impersonation. Do not copy secret values into repository secrets, local files, or long-lived environment variables unless there is no better option.

Data-Engineering Scenarios

ScenarioCorrect patternWhy it worksWhat to avoid
Cloud Run job reads one API tokenSecret-scope accessor on one secretKeeps blast radius to one containerProject-wide accessor on every secret
Airflow needs connection strings and variablesSecret Manager backend with per-secret IAMRemoves sensitive values from the Airflow metadata DBStoring passwords directly in Airflow connections UI
GitHub Actions deploys and reads deployment metadataWIF for Google auth, Secret Manager only for non-Google secrets if truly neededKeeps CI keyless for Google accessJSON service-account keys in GitHub secrets
Regional workload keeps credentials in europe-west1Regional secret plus regional endpoint operationsAligns secret residency with the workload and data boundaryAssuming --location alone is enough in the CLI
Rotating a vendor token after suspected exposureAdd or promote new version, repoint alias, disable old version, watch destroy TTLGives a rollback window and clear current-state metadataEditing a value in place or deleting the only known-good version immediately

Troubleshooting and Incident Response

SymptomLikely causeFirst checkSafe next action
Regional secret commands fail with INVALID_ARGUMENTMissing regional endpoint overrideCheck api_endpoint_overrides/secretmanagerSet the regional endpoint, perform the operation, then unset it
Secret access works as a human but not as a workloadSecret-scope IAM missing for the workload principalgcloud secrets get-iam-policyAdd roles/secretmanager.secretAccessor on the secret only
Alias resolves but access still failsAlias points to a disabled versiongcloud secrets versions list and describeRepoint or remove the alias
CMEK-backed secret create/access failsSecret Manager service agent lacks KMS rightsgcloud kms keys get-iam-policyGrant roles/cloudkms.cryptoKeyEncrypterDecrypter to the service agent
Secret was disabled or scheduled for destruction too earlyRotation runbook skipped the recovery windowgcloud secrets versions listStop promoting the broken alias, restore from a surviving version if possible, and rebuild the runbook

Quick Reference

NeedUseAvoid
One workload reads one secretSecret-scope roles/secretmanager.secretAccessorProject-wide accessor if not required
Stable consumer referenceAlias such as currentHardcoded version number everywhere
Residency boundaryRegional secret plus regional endpoint overrideAssuming automatic replication and regional requirements are equivalent
Customer key controlSecret Manager CMEK with service-agent KMS grantEnabling CMEK without validating KMS IAM
Rotation safetyDisable old version, keep destroy TTL, then remove stale aliasesImmediate irreversible destroy of the last known-good version

GCP Secrets Management References

Graph View

Backlinks