03 - Cloud Run Jobs vs Services

19 min read

Cloud Run Jobs vs Services

Quote

“Functions are the verbs of serverless; containers are the nouns. You need both parts of speech to write a complete sentence.”

— Ben Kehoe, iRobot cloud robotics engineer

Archived demo boundary

The original Serverless demo project used for this folder no longer exists. The job names, execution rows, and europe-west1 examples below are preserved as archived operator reference rather than current Cloud Run inventory, and this refresh does not rerun any Cloud Run commands.

The edits below focus on current product behavior from Cloud Run documentation, especially pricing-mode differences, startup CPU boost, sidecar billing constraints, and the newer worker-pool option for continuous background work.

Jobs vs Services Overview

Cloud Run offers two resource types: Jobs for batch workloads that run to completion, and Services for HTTP-driven workloads that stay listening. The choice determines the lifecycle, billing model, scaling behavior, and trigger mechanism.

FeatureCloud Run ServiceCloud Run Job
LifecycleRuns continuously, listens for HTTPRuns to completion, then exits
TriggerHTTP request, Pub/Sub push, EventarcManual, Cloud Scheduler, Airflow, Workflows
Scaling0 to 1,000 instances based on trafficFixed task count with configurable parallelism
Timeout60 minutes max (per request)168 hours / 7 days max (per task)
ConcurrencyUp to 1,000 concurrent requests per instanceOne task per container instance
BillingPer-request + vCPU-seconds + memory while servingvCPU-seconds + memory for the full execution duration
Cold startMitigated with min-instancesAlways cold starts (no warm pool)
Use caseAPIs, webhooks, dashboards, event receiversETL stages, data processing, batch scoring, migrations

When to use Jobs vs Services vs Cloud Functions

  • Cloud Run Jobs — batch workloads that run to completion: ETL stages, data exports, model scoring, database migrations. Best when you need custom containers, long timeouts (up to 7 days), or parallel task fan-out.
  • Cloud Run Services — request-driven workloads: REST APIs, webhook receivers, Pub/Sub push endpoints, dashboards. Best when you need auto-scaling to zero, HTTP routing, or traffic splitting.
  • Cloud Functions — lightweight event-driven glue: file upload triggers, Pub/Sub message handlers, simple transformations under 60 minutes. Best when you want zero infrastructure management and the function fits a single file. Cloud Functions (2nd gen) runs on Cloud Run under the hood.

Continuous workers are now a third container option

Cloud Run Worker Pools exist for long-running background workers that do not serve HTTP traffic and do not naturally fit run-to-completion job semantics. Use Jobs when the unit of work should finish and exit, Services when requests drive scaling, and Worker Pools when you need continuously running pull workers or similar background daemons.


flowchart TD
    A[New workload] --> B{Runs to completion?}
    B -->|Yes| C{Needs custom container<br/>or >60 min timeout?}
    B -->|No| D{Needs custom container<br/>or concurrency control?}
    C -->|Yes| E[Cloud Run Job]
    C -->|No| F[Cloud Function]
    D -->|Yes| G[Cloud Run Service]
    D -->|No| F

    style E fill:#292e42,stroke:#7aa2f7,color:#c0caf5
    style G fill:#292e42,stroke:#7aa2f7,color:#c0caf5
    style F fill:#292e42,stroke:#bb9af7,color:#c0caf5

Cloud Run Jobs

Cloud Run Jobs execute a container image to completion and then exit. A job definition specifies the container image, resource limits, environment variables, and retry policy. Each time a job is triggered, Cloud Run creates an execution — a single run of the job that can contain one or more parallel tasks. For infrastructure-as-code deployment, cloud-run provides the Terraform resource definitions.

Parallel task execution

A single job execution can run multiple identical tasks in parallel using the --tasks and --parallelism flags. Each task gets an index via the CLOUD_RUN_TASK_INDEX environment variable (0-based) and the total count via CLOUD_RUN_TASK_COUNT. Wall time scales inversely with worker count: wall_time ≈ total_cpu_seconds / parallelism. Maximum: 10,000 tasks per execution.

Archived execution examples

The job listings and execution tables in this section are historical examples from the removed project. Read them as shape-of-output references only and treat the commands themselves as reusable patterns.

gcloud | List and describe jobs

These commands show all jobs in a region and retrieve the full configuration of a specific job.

List all jobs in a region

Lists all Cloud Run Jobs in the specified region, showing the job name, region, and last execution status.

gcloud run jobs list --region=europe-west1
   JOB                        REGION        LAST RUN STATUS  EXECUTED AT
✔  data-pipeline-pipeline     europe-west1  Succeeded        2026-04-04T08:15:00Z
✔  data-export-daily          europe-west1  Succeeded        2026-04-04T06:00:00Z
✗  data-pipeline-backfill     europe-west1  Failed           2026-04-03T22:30:00Z

Describe a job’s configuration

Returns the full configuration of a job including image, resource limits, environment variables, service account, and retry policy.

gcloud run jobs describe data-pipeline-pipeline --region=europe-west1

gcloud | Execute a job

Triggering a job creates a new execution. The container starts, runs, and exits. The execution inherits the job’s configuration unless overridden at execution time.

Execute with default configuration

gcloud run jobs execute data-pipeline-pipeline --region=europe-west1

Execute with argument overrides

Pass comma-separated arguments to the container’s ENTRYPOINT. This enables running a specific pipeline stage or index without creating a separate job definition for each variant.

gcloud run jobs execute data-pipeline-pipeline --region=europe-west1 \
  --args="--stage,gold,--index,market_index"

Execute with environment variable overrides

Override environment variables at execution time without modifying the job definition. Useful for ad-hoc runs with different parameters.

gcloud run jobs execute data-pipeline-pipeline --region=europe-west1 \
  --update-env-vars="LOG_LEVEL=DEBUG,DRY_RUN=true"

gcloud | Monitor job executions

After triggering a job, monitor its progress through execution listings and logs. Job logs are written to Cloud Logging with the resource type cloud_run_job and can be queried from the Cloud Logging console.

List recent executions

Shows the execution name, status (Succeeded/Failed/Running), start time, and duration for a given job.

gcloud run jobs executions list --job=data-pipeline-pipeline \
  --region=europe-west1 --limit=5
   EXECUTION                                  STATUS     START TIME                DURATION
✔  data-pipeline-pipeline-x4k2m              Succeeded  2026-04-04T08:15:00Z      4m12s
✔  data-pipeline-pipeline-r9j1n              Succeeded  2026-04-04T04:15:00Z      3m58s
✗  data-pipeline-pipeline-w7p3q              Failed     2026-04-03T22:30:00Z      1m03s
✔  data-pipeline-pipeline-t2m8v              Succeeded  2026-04-03T16:15:00Z      4m05s
✔  data-pipeline-pipeline-k5n6r              Succeeded  2026-04-03T08:15:00Z      3m47s

View execution logs

Streams the logs for a specific execution. Replace the execution name with the value from the execution list.

gcloud run jobs executions logs data-pipeline-pipeline-x4k2m \
  --region=europe-west1

gcloud | Update job configuration

Updates the job definition for future executions. Changes take effect on the next execution — running executions are not affected. The --memory flag sets the RAM limit (128Mi to 32Gi), --cpu sets CPU allocation (1, 2, 4, 6, or 8 vCPUs for jobs), --task-timeout is the maximum execution time before a forced kill, and --max-retries controls automatic retry on failure (0 = no retry).

gcloud run jobs update data-pipeline-pipeline --region=europe-west1 \
  --image=europe-west1-docker.pkg.dev/data-platform-prod/data-pipeline/data-pipeline:latest \
  --memory=2Gi \
  --cpu=1 \
  --task-timeout=30m \
  --set-env-vars="DB_HOST=10.132.0.2,DB_NAME=data-pipeline,LOG_LEVEL=INFO" \
  --max-retries=1

gcloud | Delete a job

Permanently removes a job definition and all its execution history. This cannot be undone — the execution logs remain in Cloud Logging but the job resource is deleted.

gcloud run jobs delete data-pipeline-pipeline --region=europe-west1

Cloud Run Jobs flag reference

FlagApplies toSyntaxDescription
--regionAll commands--region=europe-west1GCP region for the job (required)
--imagecreate, update--image=REGISTRY/IMAGE:TAGContainer image to run
--memorycreate, update--memory=2GiMemory limit per task (128Mi to 32Gi)
--cpucreate, update--cpu=1vCPU allocation per task (1, 2, 4, 6, or 8 for jobs)
--task-timeoutcreate, update--task-timeout=30mMax duration per task before forced kill (up to 168h)
--max-retriescreate, update--max-retries=1Retry count on task failure (0 = no retry, max 10)
--taskscreate, update--tasks=10Number of parallel tasks per execution (max 10,000)
--parallelismcreate, update--parallelism=5Max tasks running concurrently (0 = all at once)
--argsexecute--args="--stage,gold"Comma-separated args passed to ENTRYPOINT
--update-env-varsexecute--update-env-vars="K=V"Override env vars for this execution only
--set-env-varscreate, update--set-env-vars="K=V,K2=V2"Set environment variables on the job definition
--set-secretscreate, update--set-secrets="ENV=SECRET:VERSION"Mount Secret Manager secrets as env vars
--service-accountcreate, update--service-account=SA@PROJECT.iamService account the job runs as
--vpc-connectorcreate, update--vpc-connector=CONNECTORServerless VPC Access connector for private networking
--limitlist, executions list--limit=5Maximum number of results to return
--formatAll commands--format=jsonOutput format: json, yaml, table, value

Pipeline Architecture

Each pipeline stage runs as an independent Cloud Run Job execution. Airflow orchestrates the sequence using task dependencies. This architecture allows individual stages to be retried, redeployed, or replaced without affecting the others. For CI/CD automation that builds and deploys these containers via Workload Identity, see github-actions-ci-cd.


flowchart TD
    A[Airflow DAG] --> B[gcloud run jobs execute<br/>--args='--stage,bronze']
    A --> C[gcloud run jobs execute<br/>--args='--stage,silver']
    A --> D[gcloud run jobs execute<br/>--args='--stage,gold']

    B --> E[(GCS<br/>raw files)]
    C --> E
    C --> F[(BigQuery<br/>staging)]
    D --> F
    D --> G[(BigQuery<br/>production)]

    B:::jobNode
    C:::jobNode
    D:::jobNode

    classDef jobNode fill:#292e42,stroke:#7aa2f7,color:#c0caf5

Three execution models for Cloud Run Jobs

  • Tool/script jobs — on-demand, triggered manually for ad-hoc tasks (archive logs, export a table to CSV). Output artifacts go to GCS.
  • Scheduled jobs — clock-driven via Cloud Scheduler with configurable retry policy and max execution window. If the window is exceeded, the job is terminated and restarted.
  • Array jobs — fan-out parallelism using --tasks and --parallelism. Wall time scales inversely with worker count: wall_time ≈ total_cpu_seconds / number_of_workers.

Cold Start and Performance

The first execution after a period of inactivity takes longer because Cloud Run needs to pull and start the container image. For data pipelines triggered 3x/day, cold starts are a minor annoyance (seconds, not minutes) — but for latency-sensitive services, they can be a problem.

Reducing Cold Start Latency

  • Keep images small. Cold-start work begins with image pull and process startup, so image discipline is still the first lever. Use docker-compose locally to mirror the production container environment during development.
  • Multi-stage Docker builds. Build dependencies in stage 1, copy only the runtime into the final image.
  • Lazy initialization. Defer heavy imports and connection setup until the first request or task starts, not at module load time.
  • Min instances = 1 (services only): keeps one instance warm. Not applicable to Jobs — they always cold start.
  • Startup CPU boost (services only): temporarily increases CPU during startup and is often cheaper than keeping a service warm all day.
  • Instance-based billing (services only): use it when the service or its sidecars need CPU outside request handling. Request-based billing is still the default and usually the cheaper baseline.

Cost impact of always-on settings

--min-instances is not free capacity. Under request-based billing, minimum instances still create idle billable time; under instance-based billing, the entire instance lifetime is billed. The common mistake is enabling warm capacity or CPU outside requests for low-traffic services that do not actually need it.

Right-size always-on settings

Use --min-instances only for services with a measured latency objective, prefer startup CPU boost before moving to permanently warm instances, and switch to instance-based billing only when background threads or sidecars must keep running between requests. For internal batch-trigger endpoints or low-traffic webhooks, let instances scale to zero and accept the cold start.

Cloud Run pricing model

  • Services: Cloud Run now has two service billing modes. Request-based billing charges for requests plus active CPU and memory, while instance-based billing bills the full instance lifetime and is the mode to use when CPU must stay available outside request processing.
  • Jobs: Jobs bill for CPU and memory for the duration of each task execution and have no per-request charge. The current free tier for Jobs is 240,000 vCPU-seconds and 450,000 GiB-seconds per month.
  • Request-based free tier: Services on request-based billing currently include 2 million requests, 180,000 vCPU-seconds, and 360,000 GiB-seconds per month before paid usage begins.
  • Networking and discounts: Billing is rounded to 100 ms, same-region traffic to Google Cloud resources is free, and Cloud Run flexible CUDs currently discount Jobs and instance-based services more deeply than request-based services.

Environment Variables and Secrets

Pass configuration to Cloud Run Jobs and Services via environment variables. Use --set-env-vars for non-sensitive values and --set-secrets for credentials managed in Secret Manager.

Hardcoded secrets in environment variables

Never put sensitive values (database passwords, API keys, service account keys) directly in --set-env-vars. Environment variables are visible in plain text in the Cloud Console, gcloud run jobs describe output, Terraform state files, and CI/CD logs. A single leaked export can compromise production databases.

Use Secret Manager references

Reference secrets from Secret Manager using --set-secrets. The secret value is injected at runtime and never stored in the job definition. The job’s service account needs roles/secretmanager.secretAccessor on the referenced secret.

gcloud run jobs update data-pipeline-pipeline --region=europe-west1 \
  --set-secrets="DB_PASSWORD=data-pipeline-db-password:latest"

The secret value is injected as an environment variable at runtime. The container reads DB_PASSWORD like any other env var — it never sees the secret name or version, only the resolved value. For managing secrets in Secret Manager, see secrets-management.

Multi-container sidecar support

Cloud Run supports deploying multiple containers in a single service or job revision (sidecar pattern). A sidecar container runs alongside the main container, shares the same network namespace, and can share files through in-memory volumes. Common uses include logging agents, auth proxies, metrics collectors, and secret sidecars.

For services on request-based billing, sidecars only get CPU during request handling and startup. If a sidecar must keep working between requests, such as continuous metrics shipping or a long-lived proxy loop, move that service to instance-based billing or reconsider whether a worker pool is the better primitive.

Cloud Run Jobs vs Services References

Graph View

Backlinks