05 - Registry and CI

14 min read

Registry and CI

Quote

“Winning developers means earning their trust over many years through great software.”

— Mitchell Hashimoto, HashiConf talk

Artifact Registry

Artifact Registry is GCP’s container image storage service, the successor to the deprecated Container Registry. Docker images are pushed here by GitHub Actions and pulled by Cloud Run at deployment time. This section covers the Terraform resource that provisions the repository and its automated cleanup policies.

Assumed variables and prerequisites

  • var.region — GCP region (e.g., europe-west1)
  • var.project_id — GCP project ID
  • Required API: artifactregistry.googleapis.com
  • Required IAM role for the Terraform service account: roles/artifactregistry.admin

google_artifact_registry_repository

Provisions a Docker image repository in Artifact Registry. Cloud Run services and jobs pull images from this repository at deploy time, so co-locating the registry in the same region as Cloud Run minimizes pull latency.

The format argument is immutable — changing it forces resource replacement. The location and repository_id arguments are also immutable; changing either one destroys the existing repository and creates a new one.

Provision a Docker repository with keep-5 and delete-untagged cleanup policies.

resource "google_artifact_registry_repository" "data-pipeline" {
  location      = var.region
  repository_id = "data-pipeline"
  format        = "DOCKER"
 
  cleanup_policies {
    id     = "keep-latest-5"
    action = "KEEP"
 
    most_recent_versions {
      keep_count = 5
    }
  }
 
  cleanup_policies {
    id     = "delete-untagged"
    action = "DELETE"
 
    condition {
      tag_state = "UNTAGGED"
    }
  }
}
ArgumentRequiredDescription
locationYesGCP region for image storage. Co-locate with Cloud Run to minimize pull latency. Immutable — changing forces replacement.
repository_idYesUnique repository name within the project. Forms the registry path: <region>-docker.pkg.dev/<project>/<repository_id>/. Immutable — changing forces replacement.
formatYesRepository format: DOCKER, MAVEN, NPM, PYTHON, APT, YUM, GO. Immutable.
cleanup_policiesNoAutomated lifecycle rules for image retention and deletion. Multiple policies can coexist.
cleanup_policy_dry_runNoSet to true to log which images would be affected by cleanup policies without actually deleting them. Use this to validate policies before enforcement. Default: false.

Force-replacement triggers

Changing location, repository_id, or format destroys the repository and all its images, then creates a new empty repository. This is permanent data loss — deleted images cannot be recovered.

Protect with lifecycle rules

Add lifecycle { prevent_destroy = true } to production registries. If a replacement is genuinely needed, first push all images to the new repository, update all Cloud Run references, then remove the old resource from state with terraform state rm before deleting it from config.

Cleanup Policies

Two policies work together to prevent unbounded storage growth. Without cleanup policies, every CI push accumulates permanently.

PolicyActionRuleEffect
keep-latest-5KEEPkeep_count = 5Retains the 5 most recent image versions per tag. Older versions become candidates for deletion.
delete-untaggedDELETEtag_state = UNTAGGEDDeletes images that have no tag (e.g., after a newer image takes the latest tag, the old one becomes untagged). Prevents orphaned layers from consuming storage.

Validate before enforcing

Set cleanup_policy_dry_run = true when first adding cleanup policies. Artifact Registry will log which images would be affected in Cloud Logging without actually deleting them. Once satisfied, set it back to false to enable enforcement. With daily deployments and a 5-version retention window, storage stays bounded to approximately 5× the image size per repository.

Lifecycle Meta-Arguments

Production registries should be protected from accidental deletion. The prevent_destroy meta-argument causes Terraform to error if a plan would destroy the repository.

Protect the production registry from accidental deletion.

resource "google_artifact_registry_repository" "data-pipeline" {
  location      = var.region
  repository_id = "data-pipeline"
  format        = "DOCKER"
 
  lifecycle {
    prevent_destroy = true
  }
}

Registry Path Structure

The locals.registry value in run.tf computes the full registry path from the repository attributes:

europe-west1-docker.pkg.dev/data-platform-prod/data-pipeline

Images are referenced as:

  • europe-west1-docker.pkg.dev/data-platform-prod/data-pipeline/pipeline:latest
  • europe-west1-docker.pkg.dev/data-platform-prod/data-pipeline/dashboard:latest

Import existing Artifact Registry repository

To bring an existing repository under Terraform management:

CLI import (all Terraform versions):

terraform import google_artifact_registry_repository.data-pipeline projects/data-platform-prod/locations/europe-west1/repositories/data-pipeline

Declarative import (Terraform 1.5+):

import {
  to = google_artifact_registry_repository.data-pipeline
  id = "projects/data-platform-prod/locations/europe-west1/repositories/data-pipeline"
}

CI/CD Service Account

A dedicated service account for GitHub Actions with narrowly scoped permissions. This account can push Docker images and deploy Cloud Run revisions, but cannot access databases, read secrets, or modify IAM policies. Its credentials are stored as a GitHub Actions secret (GCP_SA_KEY).

Assumed variables and dependencies

  • var.project_id — GCP project ID
  • google_service_account.pipeline — Pipeline workload SA (defined in iam-and-secrets)
  • google_service_account.dashboard — Dashboard workload SA (defined in iam-and-secrets)
  • Required API: iam.googleapis.com
  • Required IAM role for the Terraform service account: roles/iam.serviceAccountAdmin

google_service_account | CI Runner

Creates the service account identity that GitHub Actions authenticates as when deploying to GCP. The account_id is immutable — changing it forces replacement, which invalidates any existing keys or Workload Identity bindings.

Create the GitHub Actions CI/CD service account.

resource "google_service_account" "ci" {
  account_id   = "data-pipeline-ci"
  display_name = "the data pipeline project CI/CD (GitHub Actions)"
}
ArgumentRequiredDescription
account_idYesUnique identifier within the project. Forms the email: <account_id>@<project>.iam.gserviceaccount.com. Immutable — changing forces replacement.
display_nameNoHuman-readable label shown in the GCP console. Mutable.

google_project_iam_member | CI Registry and Cloud Run Access

Grants project-level IAM roles to the CI service account. Each google_project_iam_member resource adds a single role binding without overwriting existing bindings for that role (unlike google_project_iam_binding which is authoritative).

Grant Artifact Registry write access and Cloud Run deploy access to the CI account.

resource "google_project_iam_member" "ci_registry" {
  project = var.project_id
  role    = "roles/artifactregistry.writer"
  member  = "serviceAccount:${google_service_account.ci.email}"
}
 
resource "google_project_iam_member" "ci_run" {
  project = var.project_id
  role    = "roles/run.developer"
  member  = "serviceAccount:${google_service_account.ci.email}"
}
ArgumentRequiredDescription
projectYesGCP project to bind the role in.
roleYesIAM role to grant. Changing forces replacement (removes old binding, adds new one).
memberYesIdentity receiving the role. Format: serviceAccount:<email>. Changing forces replacement.
RoleWhat It Allows
------
roles/artifactregistry.writerPush (write) Docker images to Artifact Registry. Cannot delete images or modify repository settings.
roles/run.developerDeploy new revisions to Cloud Run services and jobs. Cannot modify IAM or networking.

google_service_account_iam_member | Act-As Bindings

Grants the CI service account permission to act as the workload service accounts when deploying Cloud Run services and jobs. When GitHub Actions deploys a Cloud Run revision, it must specify which service account the workload runs as — the roles/iam.serviceAccountUser role on the target SA authorizes this without granting the CI account the target SA’s own permissions.

Allow the CI account to act as the pipeline SA and dashboard SA when deploying Cloud Run workloads.

resource "google_service_account_iam_member" "ci_act_as_pipeline" {
  service_account_id = google_service_account.pipeline.name
  role               = "roles/iam.serviceAccountUser"
  member             = "serviceAccount:${google_service_account.ci.email}"
}
 
resource "google_service_account_iam_member" "ci_act_as_dashboard" {
  service_account_id = google_service_account.dashboard.name
  role               = "roles/iam.serviceAccountUser"
  member             = "serviceAccount:${google_service_account.ci.email}"
}
ArgumentRequiredDescription
service_account_idYesThe target service account (the one being “acted as”). Use the name attribute (projects/…/serviceAccounts/…) not the email.
roleYesIAM role to grant on the target service account.
memberYesThe identity receiving the role. Format: serviceAccount:<email>.

Least privilege chain

The CI account can push images and update deployments, but it cannot access the database, read secrets, or trigger pipeline runs. It can only assign existing service accounts to Cloud Run workloads. See iam-and-secrets for the full IAM design and service-accounts-and-iam for GCP IAM fundamentals.

Verification

Use gcloud commands to verify the Terraform-provisioned resources match expectations.

List repositories

List all Artifact Registry repositories in the deployment region.

gcloud artifacts repositories list --location=europe-west1

List Docker images

List all Docker image versions stored in the data-pipeline repository.

gcloud artifacts docker images list europe-west1-docker.pkg.dev/data-platform-prod/data-pipeline

List tags for pipeline image

Show all tags applied to the pipeline image.

gcloud artifacts docker tags list europe-west1-docker.pkg.dev/data-platform-prod/data-pipeline/pipeline

List tags for dashboard image

Show all tags applied to the dashboard image.

gcloud artifacts docker tags list europe-west1-docker.pkg.dev/data-platform-prod/data-pipeline/dashboard

Verify CI service account roles

Show IAM roles bound to the CI service account at the project level.

gcloud projects get-iam-policy data-platform-prod --flatten="bindings[].members" --filter="bindings.members:data-pipeline-ci@" --format="table(bindings.role)"

GitHub Actions Workflow Integration

The CI service account authenticates GitHub Actions workflows with GCP. There are two authentication methods: JSON key (legacy) and Workload Identity Federation (modern, keyless).

JSON Key Authentication (Legacy)

The CI service account’s JSON key is stored as a GitHub Actions secret and passed to the google-github-actions/auth action. This method works but carries security risks — the key is a long-lived credential that must be manually rotated and can be exfiltrated if the repository is compromised.

Authenticate GitHub Actions with a JSON service account key (legacy).

steps:
  - uses: google-github-actions/auth@v2
    with:
      credentials_json: ${{ secrets.GCP_SA_KEY }}

Long-lived credentials

JSON service account keys do not expire automatically. If the key leaks (e.g., through a compromised CI log, a forked repository, or a misconfigured workflow), an attacker gains persistent access until the key is manually revoked. Key rotation is a manual process with no built-in enforcement.

Migrate to Workload Identity Federation

Workload Identity Federation eliminates long-lived keys entirely. GitHub Actions exchanges a short-lived OIDC token for temporary GCP credentials — no secrets to rotate, no keys to leak. See the Workload Identity Federation section below.

GitHub Actions Secrets

These secrets are configured in the GitHub repository settings and injected into workflow runs.

Secret NameWhat It ContainsUsed By
GCP_SA_KEYService account key JSON (entire file contents)google-github-actions/auth for GCP authentication
DD_API_KEYDatadog API keyPipeline containers for APM/log shipping
DB_PASSWORDDatabase SA passwordPipeline and dashboard containers

Workload Identity Federation (WIF) allows GitHub Actions to authenticate with GCP without storing any long-lived credentials. GitHub’s OIDC provider issues a short-lived token, which GCP exchanges for temporary service account credentials scoped to a single workflow run.

Create a Workload Identity pool and OIDC provider for GitHub Actions.

resource "google_iam_workload_identity_pool" "github" {
  workload_identity_pool_id = "github-actions"
  display_name              = "GitHub Actions"
}
 
resource "google_iam_workload_identity_pool_provider" "github" {
  workload_identity_pool_id          = google_iam_workload_identity_pool.github.workload_identity_pool_id
  workload_identity_pool_provider_id = "github-oidc"
  display_name                       = "GitHub OIDC"
 
  attribute_mapping = {
    "google.subject"       = "assertion.sub"
    "attribute.repository" = "assertion.repository"
  }
 
  attribute_condition = "assertion.repository == 'your-org/your-repo'"
 
  oidc {
    issuer_uri = "https://token.actions.githubusercontent.com"
  }
}

Allow the CI SA to be impersonated via Workload Identity Federation.

resource "google_service_account_iam_member" "ci_wif" {
  service_account_id = google_service_account.ci.name
  role               = "roles/iam.workloadIdentityUser"
  member             = "principalSet://iam.googleapis.com/${google_iam_workload_identity_pool.github.name}/attribute.repository/your-org/your-repo"
}

The GitHub Actions workflow then authenticates without any JSON key:

Authenticate GitHub Actions via Workload Identity Federation (keyless).

steps:
  - uses: google-github-actions/auth@v2
    with:
      workload_identity_provider: projects/<project-number>/locations/global/workloadIdentityPools/github-actions/providers/github-oidc
      service_account: data-pipeline-ci@data-platform-prod.iam.gserviceaccount.com

WIF benefits

  • No secrets to rotate — tokens are issued per workflow run and expire automatically
  • Repository-scoped — the attribute_condition restricts authentication to a specific GitHub repository
  • Audit trail — each token exchange is logged in Cloud Audit Logs with the originating repository and workflow

For the full GitHub Actions CI/CD workflow setup, see github-actions-ci-cd. For Docker image build and push operations, see image-management.

CI/CD Pipeline Architecture

This diagram shows how the Terraform resources in this file connect to form the deployment pipeline — from code push through image storage to Cloud Run deployment.


flowchart LR
    A[Git Push] --> B[GitHub Actions]
    B -->|auth@v2<br>SA Key or WIF| C[CI Service Account]
    C -->|artifactregistry.writer| D[Artifact Registry]
    C -->|run.developer| E[Cloud Run Deploy]
    C -->|iam.serviceAccountUser| F[Workload SAs]
    D -->|docker pull| E
    G[Cleanup Policies] -->|keep 5 / delete untagged| D
    F -->|pipeline SA| H[Cloud Run Job]
    F -->|dashboard SA| I[Cloud Run Service]

Terraform chapter:

  • iam-and-secrets — full IAM design and the workload service accounts this CI account acts as
  • cloud-run — Cloud Run services and jobs that pull images from this registry

GCP services (Folder 06):

CI/CD and Docker:

Terraform Registry and CI References

Graph View

Backlinks