01 - Pub/Sub Topics and Subscriptions

17 min read

Pub/Sub Topics and Subscriptions

Quote

“The key in making great and growable systems is much more to design how its modules communicate rather than what their internal properties and behaviors should be.”

— Alan Kay, The Early History of Smalltalk (1993)

Knowledge-only refresh

The original Serverless demo project used for this folder no longer exists. This note keeps the generic gcloud examples and sample outputs as operator reference, but no Pub/Sub topics, subscriptions, push endpoints, or dead-letter paths were recreated during this refresh.

The edits below only correct product behavior from current Pub/Sub documentation, including immutable subscription filters, export-subscription guidance, and current storage-pricing boundaries.

Topic Management

A topic is the named channel that producers publish messages to. Topics are regional resources — messages published to a topic in us-central1 are stored in that region. Multiple subscriptions can attach to the same topic, each receiving an independent copy of every message (fan-out pattern). The maximum message size is 10 MB per message.

gcloud | Create and manage topics

Use gcloud pubsub topics create to create a new topic. A topic must exist before any subscription can be attached to it or any message published. Topics can optionally enforce schema validation (Avro or Protocol Buffers) to reject malformed messages at publish time.

gcloud | Create a topic

Create a named topic in the current project. The topic name must be unique within the project and can contain letters, numbers, hyphens, and underscores.

gcloud pubsub topics create pipeline-events
Created topic [projects/my-project/topics/pipeline-events].

gcloud | Create a topic with message retention

By default, topics do not retain messages — once delivered to all subscriptions, the message is eligible for deletion. Enabling topic-level message retention stores messages for the specified duration, allowing subscriptions created later to replay historical messages.

gcloud pubsub topics create pipeline-events-retained \
  --message-retention-duration=7d
Created topic [projects/my-project/topics/pipeline-events-retained].

gcloud | List topics

List all topics in the current project. Useful for verifying topic creation or auditing existing topics.

gcloud pubsub topics list --format="table(name)"
NAME
projects/my-project/topics/pipeline-events
projects/my-project/topics/pipeline-events-retained
projects/my-project/topics/pipeline-events-dead-letter
FlagSyntaxDescription
--message-retention-duration--message-retention-duration=7dRetain messages at the topic level for replay (max 31 days)
--schema--schema=my-schemaBind an Avro or Protocol Buffer schema for publish-time validation
--message-encoding--message-encoding=JSONEncoding for schema-validated messages (JSON or BINARY)
--labels--labels=env=prod,team=dataKey-value labels for cost tracking and resource organization
--kms-key--kms-key=projects/.../cryptoKeys/keyCustomer-managed encryption key (CMEK) for message encryption at rest

Subscription Management

Subscriptions are the delivery mechanisms that connect consumers to topics. Each subscription receives an independent copy of every message published to its topic. Pub/Sub supports four subscription types: pull (consumer fetches messages), push (Pub/Sub delivers to an HTTP endpoint), BigQuery (Pub/Sub writes directly to a BigQuery table), and Cloud Storage (Pub/Sub writes to GCS buckets). BigQuery and Cloud Storage subscriptions are export subscriptions: they remove subscriber code when the destination only needs direct durable writes, but Dataflow or a custom consumer is still the better fit once you need joins, windowing, aggregation, or rich transformation. A single topic can have up to 10,000 subscriptions.

gcloud | Create a pull subscription

Pull subscriptions require the consumer to actively fetch messages. The consumer controls the rate of processing, making pull the default choice for batch pipelines and variable-rate workloads. The --ack-deadline gives the consumer a window to process and acknowledge each message before Pub/Sub redelivers it. The --message-retention-duration keeps messages in the subscription backlog for the specified period, enabling replay.

gcloud | Create a pull subscription with retention

Create a pull subscription attached to an existing topic. The ack deadline of 60 seconds gives the consumer one minute to process each message before redelivery. The 7-day retention enables replaying messages for debugging.

gcloud pubsub subscriptions create pipeline-sub \
  --topic=pipeline-events \
  --ack-deadline=60 \
  --message-retention-duration=7d
Created subscription [projects/my-project/subscriptions/pipeline-sub].

At-Least-Once Delivery

Pub/Sub guarantees at-least-once delivery: every message will be delivered at least once, but may be delivered more than once. The --ack-deadline determines how long the consumer has to process and acknowledge a message before Pub/Sub considers it unacknowledged and redelivers it. Set the deadline to exceed your maximum expected processing time plus margin. The default is 10 seconds; the maximum is 600 seconds.

gcloud | Create a pull subscription with attribute filtering

Subscription-level attribute filters allow the subscription to receive only messages matching a specific condition. The filter is evaluated server-side — non-matching messages are automatically acknowledged and never delivered to the consumer. This enables a single topic to serve multiple consumers with different filtering criteria, without any client-side logic.

gcloud pubsub subscriptions create pipeline-gold-sub \
  --topic=pipeline-events \
  --ack-deadline=60 \
  --message-filter='attributes.stage = "gold"'
Created subscription [projects/my-project/subscriptions/pipeline-gold-sub].

Filters are immutable and still bill throughput

Pub/Sub lets pull and push subscriptions filter on message attributes, but you cannot edit the filter on an existing subscription. The safe change path is to snapshot the old subscription, create a new one with the new filter, then seek the new subscription to that snapshot.

Non-matching messages are automatically acknowledged for that subscription, but Pub/Sub throughput charges still apply to those filtered messages.

Subscription Expiration

Subscriptions with no subscriber activity (pull, push delivery, or message backlog) for 31 days are automatically deleted by default. This can silently break pipelines that process data on an infrequent schedule (e.g., monthly batch jobs).

Set Expiration Policy Explicitly

Set --expiration-period=never on subscriptions that must persist regardless of activity. For time-limited subscriptions, set an explicit duration. Always verify subscription existence before publishing to prevent silent message loss.

FlagSyntaxDescription
--topic--topic=my-topicTopic to subscribe to (required)
--ack-deadline--ack-deadline=60Seconds before unacknowledged messages are redelivered (default: 10, max: 600)
--message-retention-duration--message-retention-duration=7dHow long unacknowledged messages are retained (default: 7d, max: 31d)
--expiration-period--expiration-period=neverAuto-delete subscription after inactivity (default: 31d, never to disable)
--message-filter--message-filter='attributes.key = "val"'Server-side attribute filter expression (immutable after creation)
--enable-exactly-once-delivery--enable-exactly-once-deliveryEnable exactly-once delivery for pull subscriptions in a single region
--retain-acked-messages--retain-acked-messagesKeep acknowledged messages for replay via seek
--labels--labels=env=prodKey-value labels for cost tracking

gcloud | Create a push subscription

Push subscriptions have Pub/Sub deliver messages as HTTP POST requests to an endpoint. The endpoint must return a 2xx status to acknowledge the message; any other response triggers redelivery with configurable exponential backoff. Push is the natural fit for event-driven architectures where a Cloud Run service or Cloud Function processes messages as they arrive.

gcloud | Create a push subscription to a Cloud Run endpoint

Create a push subscription that delivers messages to a Cloud Run service URL. Pub/Sub authenticates the push request using an OIDC token from the specified service account, which the receiving service validates.

gcloud pubsub subscriptions create pipeline-push \
  --topic=pipeline-events \
  --push-endpoint=https://my-service-xyz.run.app/pubsub \
  --push-auth-service-account=pubsub-invoker@my-project.iam.gserviceaccount.com
Created subscription [projects/my-project/subscriptions/pipeline-push].

gcloud | Configure push retry policy

By default, push subscriptions retry immediately on failure. For transient errors (HTTP 429, 5xx), configure exponential backoff to avoid overwhelming the endpoint. The minimum backoff is the initial delay; the maximum backoff caps the exponential growth.

gcloud pubsub subscriptions update pipeline-push \
  --min-retry-delay=10s \
  --max-retry-delay=600s
Updated subscription [projects/my-project/subscriptions/pipeline-push].
FlagSyntaxDescription
--push-endpoint--push-endpoint=https://...HTTPS URL that receives POST requests with messages
--push-auth-service-account--push-auth-service-account=sa@project.iam.gserviceaccount.comService account for OIDC authentication on push requests
--push-auth-token-audience--push-auth-token-audience=https://...Audience claim for the OIDC token (defaults to push endpoint URL)
--min-retry-delay--min-retry-delay=10sMinimum backoff delay for push delivery retries
--max-retry-delay--max-retry-delay=600sMaximum backoff delay for push delivery retries

gcloud | Configure dead letter topics

Dead letter topics capture messages that repeatedly fail processing. After a configurable number of delivery attempts, Pub/Sub stops trying to deliver the message to the main subscription and routes it to the dead letter topic instead. Without a dead letter topic, a single unprocessable message — a malformed payload, a persistent consumer bug, or an oversized record — retries indefinitely and can block all subsequent message processing.

gcloud | Create a dead letter topic

Create a dedicated topic to receive messages that exhaust their delivery attempts. A separate subscription on the dead letter topic allows you to inspect, debug, and optionally reprocess failed messages.

gcloud pubsub topics create pipeline-events-dead-letter
Created topic [projects/my-project/topics/pipeline-events-dead-letter].

gcloud | Attach dead letter policy to subscription

Update an existing subscription to forward messages after a maximum number of delivery attempts. The Pub/Sub service account must have roles/pubsub.publisher on the dead letter topic and roles/pubsub.subscriber on the source subscription for forwarding to work.

gcloud pubsub subscriptions update pipeline-sub \
  --dead-letter-topic=pipeline-events-dead-letter \
  --max-delivery-attempts=5
Updated subscription [projects/my-project/subscriptions/pipeline-sub].

Monitor Dead-Lettered Messages

Use the Cloud Monitoring metric pubsub.googleapis.com/subscription/dead_letter_message_count to alert when messages are being dead-lettered. Additionally, monitor oldest_unacked_message_age on the source subscription — a rising value above your ack deadline threshold often signals that poison-pill messages are causing repeated NACKs before they reach the dead letter topic.

FlagSyntaxDescription
--dead-letter-topic--dead-letter-topic=my-dlq-topicTopic to receive messages that exceed max delivery attempts
--max-delivery-attempts--max-delivery-attempts=5Number of delivery attempts before forwarding to dead letter (min: 5, max: 100)
--clear-dead-letter-policy--clear-dead-letter-policyRemove the dead letter policy from a subscription

Pull vs Push Delivery Models

Choosing between pull and push delivery depends on the consumer architecture. Pull gives the consumer full control over message rate and backpressure — the consumer calls pull() when ready. Push offloads delivery timing to Pub/Sub, which sends messages as HTTP POST requests to an endpoint. For data engineering workloads, pull is the default; push suits stateless HTTP services like Cloud Run or Cloud Functions.

FeaturePullPush
Consumer controls rateYesNo (Pub/Sub controls delivery rate)
Works with Cloud RunBoth — but push is simplerYes (native trigger)
Works with offline consumersYes (messages accumulate in backlog)No (requires always-on HTTP endpoint)
BackpressureNatural (consumer stops pulling)Configurable exponential backoff retry
AuthenticationConsumer authenticates to Pub/SubPub/Sub authenticates to endpoint via OIDC
Best forBatch pipelines, controlled throughputEvent-driven triggers, serverless

Pull vs Push vs BigQuery Subscription

Pull — the consumer controls the pace. Best for batch pipelines, variable-rate processing, and consumers that need backpressure control. Use when the consumer is a Cloud Run Job, Dataflow pipeline, or any long-running process.

Push — Pub/Sub sends each message as an HTTP POST to a configured endpoint. Best for event-driven architectures where a stateless Cloud Run Service or Cloud Function processes messages as they arrive.

BigQuery subscription — Pub/Sub writes messages directly to a BigQuery table without any consumer code. Best for analytics pipelines where messages are structured data destined for BigQuery. Supports schema mapping, dead-letter handling, and uses the BigQuery Storage Write API internally, but still follows at-least-once delivery semantics. Eliminates the need for an intermediate consumer process entirely.

BigQuery Subscriptions (GA)

BigQuery subscriptions write messages directly to a BigQuery table using the Storage Write API, with no consumer process required. They are the simplest Pub/Sub → BigQuery path when the messages do not need pre-ingestion transformation, but they remain an at-least-once export mechanism rather than an exactly-once sink. Create with: gcloud pubsub subscriptions create my-bq-sub --topic=my-topic --bigquery-table=project:dataset.table.

Cloud Storage export subscriptions

Cloud Storage subscriptions are the parallel export pattern for raw event capture. Pub/Sub batches messages into objects in an existing bucket and acknowledges the source message only after the object write succeeds, which makes the feature useful for durable archiving without standing up Dataflow.

Use a Cloud Storage subscription when the destination just needs stored event files, optionally with lightweight SMT-based reshaping. If the pipeline needs cross-message aggregation, windowing, or non-trivial transformation, a Dataflow subscriber is still the better choice.


flowchart TD
    P[Producer] -->|publish| T[pipeline-events<br/>Topic]

    T -->|fan-out| S1[pipeline-sub<br/>Pull Subscription]
    T -->|fan-out| S2[pipeline-push<br/>Push Subscription]
    T -->|fan-out| S3[pipeline-gold-sub<br/>Filtered Subscription<br/>stage = gold]
    T -->|fan-out| S4[pipeline-bq-sub<br/>BigQuery Subscription]

    S1 -->|pull| C1[Batch Consumer<br/>Cloud Run Job]
    S2 -->|HTTP POST| C2[Event Handler<br/>Cloud Run Service]
    S3 -->|pull| C3[Gold Pipeline<br/>Consumer]
    S4 -->|Storage Write API| BQ[BigQuery Table]

    S1 -.->|max retries exceeded| DL[Dead Letter Topic]
    S2 -.->|max retries exceeded| DL
    S4 -.->|schema mismatch| DL

    DL --> DLS[Dead Letter<br/>Subscription]
    DLS -->|inspect + reprocess| OPS[Operations Team]

Pub/Sub Topics and Subscriptions References

Graph View

Backlinks