Skip to main content
Ask your AI

Google Pub/Sub

Server Source code Package

Subscribes to a Google Cloud Pub/Sub topic and forwards each delivered message to the walkerOS collector. The package ships two delivery models: a long-running streaming pull subscriber for container deployments, and an HTTP push webhook handler for serverless. It has decoders for JSON, text, and raw payloads, idempotent subscription provisioning with optional dead-letter wiring, and optional OIDC verification on the push handler.

The source ships inside @walkeros/server-source-gcp alongside the Cloud Functions handler; install the package once and import sourcePubSubPull or sourcePubSubPush.

Where this fits

Pub/Sub is a server source in the walkerOS flow:

Forwards each Pub/Sub message to the collector. Pull mode runs a streaming subscriber in long-lived containers; push mode receives HTTP envelopes in serverless deployments.

Installation​

npm install @walkeros/server-source-gcp

Choosing pull vs push​

AspectPull (sourcePubSubPull)Push (sourcePubSubPush)
Process modelLong-running streaming subscriberStateless HTTP request handler
HostingCloud Run with min-instances >= 1, GKE, GCE, on-premCloud Run scale-to-zero, Cloud Functions, Lambda
Idle costPay for always-on computeZero when idle
ThroughputHighest (gRPC streaming)Limited by HTTP overhead
LatencyLowest (warm stream, single-digit ms)Higher (HTTP per message, cold-start adds 100-500ms)
Auth directionSubscriber holds credentials, connects outboundPub/Sub holds credentials, posts inbound (OIDC verify)
Subscription typesubscription.type = "pull"subscription.type = "push" with pushConfig.pushEndpoint
Best fitLong-running collector services, high-volume streamsServerless walkerOS endpoints, hybrid setups

Both modes share the @walkeros/server-source-gcp package and the decoders; only pull has a setup command and SDK authentication. Pick the one that matches your hosting model.

Pull subscriber​

import { sourcePubSubPull } from '@walkeros/server-source-gcp';
import { startFlow } from '@walkeros/collector';

await startFlow({
  sources: {
    pubsub: {
      code: sourcePubSubPull,
      config: {
        settings: {
          projectId: 'YOUR_PROJECT_ID',
          subscription: 'YOUR_SUBSCRIPTION_NAME',
        },
      },
    },
  },
  destinations: {
    // your destinations
  },
});

The pull source is event-driven: consumption starts when the collector runs, and the source then opens the streaming subscription and forwards each delivered message to the collector via env.push. The source's push() called without arguments does nothing; called with a synthetic message it runs the same handler (tests, simulation). destroy() closes the subscriber gracefully, honoring shutdownTimeoutMs (default 30000).

Pull settings​

FieldTypeDefaultDescription
projectIdstring(required)GCP project id.
subscriptionstring(required)Subscription short name.
topicstring(optional)Topic short name. Setup needs it to create the subscription (and setup.createTopic to create the topic).
credentialsstring | ServiceAccountCredentials(ADC fallback)Service account JSON (parsed object or JSON string). Set on config.credentials (sibling of settings).
apiEndpointstring(SDK default)Override for the emulator (e.g. localhost:8085).
decoder'json' | 'text' | 'raw''json'How the subscriber decodes message bodies before forwarding.
flowControl{ maxMessages?, maxBytes? }{ maxMessages: 100, maxBytes: 10 MB }Subscriber pull rate (in-flight cap).
ackDeadlinenumber60Per-message ack window in seconds.
shutdownTimeoutMsnumber30000Graceful drain budget on destroy().
onPushError'nack' | 'ack''nack'Behavior when forwarding to the collector throws or fails for a retryable reason.

Push webhook​

import express from 'express';
import { Source } from '@walkeros/core';
import {
  sourcePubSubPush,
  type SourcePubSubPush,
} from '@walkeros/server-source-gcp';
import { startFlow } from '@walkeros/collector';

const { collector } = await startFlow({
  sources: {
    pubsub: {
      code: sourcePubSubPush,
      config: { settings: { decoder: 'json' } },
    },
  },
  destinations: {
    // your destinations
  },
});

const app = express();
app.use(express.json());
app.post(
  '/pubsub-push',
  Source.getSource<SourcePubSubPush.Types>(collector, 'pubsub').push,
);
app.listen(8080);

Pub/Sub posts each message envelope to the configured endpoint as a JSON body matching the Pub/Sub push spec. The handler decodes the envelope, base64-decodes the data field, runs the configured decoder, and forwards to the collector. Status codes:

  • 200: forwarded successfully, or failed for a reason a redelivery cannot fix (it is in the destination's dead-letter queue and in the collector's ERROR line).
  • 200 with dropped: true: the payload decoded to null; nothing is forwarded and nothing is logged.
  • 400: the message is permanently invalid, either a malformed envelope (missing message.data, invalid base64) or an event the pipeline rejected. A retry cannot fix either, so the answer is a nack.
  • 401: OIDC verification failed (when enabled).
  • 500: collector push failed on an otherwise valid message for a retryable reason, or threw; Pub/Sub will retry per the subscription's retry policy.

Because a permanently invalid message is nacked rather than acknowledged, Pub/Sub keeps redelivering it. Set a dead-letter policy on the subscription: that policy, not the source, is the poison-message guard.

Push settings​

Push settings: projectId (optional, not used by the handler), decoder, and:

FieldTypeDefaultDescription
verifyOidcbooleanfalseWhen true, verifies the bearer token Pub/Sub attaches to push requests.
audiencestring-Required when verifyOidc is true. Must match your endpoint URL exactly.

Authentication​

Three modes for the pull subscriber, evaluated in order:

  1. Application Default Credentials (ADC). Default. Works on GCP-native runtimes (Cloud Run, Cloud Functions, GKE) and locally with gcloud auth application-default login.
  2. Service account JSON. Pass config.credentials, either parsed or as JSON string (the source JSON-parses strings). Combine with $env.NAME to inject from an environment variable.
  3. Pre-configured client. Pass an existing PubSub SDK instance as settings.client.

For the push handler, auth is reversed: Pub/Sub holds the identity and signs requests with an OIDC token. The handler verifies the token against GCP public keys when verifyOidc: true and audience is set. Off by default because a misconfigured OIDC answers every message with 401, so opt in once your endpoint is publicly reachable.

Setup​

In a flow file the step names the pull export with "import", because the package has no default export, and the topic the subscription is created on:

"sources": {
  "pubsub": {
    "package": "@walkeros/server-source-gcp",
    "import": "sourcePubSubPull",
    "config": {
      "settings": {
        "projectId": "YOUR_PROJECT_ID",
        "subscription": "YOUR_SUBSCRIPTION_NAME",
        "topic": "YOUR_TOPIC_NAME"
      },
      "setup": true
    }
  }
}

walkeros setup source.pubsub would provision the subscription once per environment; today it cannot load the source:

walkeros setup cannot load this source today

walkeros setup source.pubsub stops with Package @walkeros/server-source-gcp has no export "sourcePubSubPull" (or has no default export without "import") and creates nothing. Create the subscription with gcloud pubsub subscriptions create or your infrastructure code. The rest of this section describes the source's own setup function.

Idempotent. Defaults: Pub/Sub's own (10-second ack deadline), project-default retention, no filter, no dead-letter policy. Drift on ackDeadlineSeconds, messageRetentionDuration, deadLetterPolicy, enableMessageOrdering, or filter logs one setup.drift warning per field ({ field, declared, actual }) on the logger the setup function receives, and never auto-mutates.

config.setup:

  • omitted or false (default): no provisioning.
  • true: provisions the subscription on settings.topic (an existing topic) with safe defaults. Without settings.topic setup creates nothing, and walkeros setup still reports setup: ok.
  • { ackDeadlineSeconds, messageRetentionDuration, filter, deadLetterPolicy, retryPolicy, ... }: object form for explicit overrides.

Optional one-shot helpers:

  • setup.createTopic: true: auto-create the topic if it does not exist (requires settings.topic). Useful when source and topic owners share a deployment.
  • setup.deadLetterPolicy.createDeadLetterTopic: true: auto-create the dead-letter topic referenced in deadLetterPolicy.deadLetterTopic.

The provisioning identity needs pubsub.subscriptions.create (and pubsub.topics.create if auto-creating). The runtime pull identity needs roles/pubsub.subscriber on the subscription.

Topic provisioning is owned by the Pub/Sub destination when the topic is the producer's responsibility; the source can also auto-create as a convenience for self-contained setups.

Decoders​

DecoderBehavior
json (default)JSON.parse(data.toString('utf8'))
textThe UTF-8 string, forwarded as data.payload (pull) or as the push value (push)
rawThe bytes, as a base64 string in data.payload (pull) or the Buffer (push)

Neither text nor raw carries an event name: without a source.before transformer that builds the event, the collector rejects it as invalid, so pull acknowledges and drops the message and push answers 400. A JSON payload of null is acknowledged and dropped with a DEBUG line only (push: 200 with dropped: true, no log line).

A decoder that throws on malformed payloads triggers onPushError (pull mode): nack redelivers, ack drops it; the decode failure is an ERROR line. In pull mode a failure a redelivery cannot fix (a missing setting, a 4xx from the vendor, a destination that is not ready) and invalid input are acknowledged: the failure is in the destination's dead-letter queue and in one ERROR line from the collector, and invalid input is counted as rejected with one ERROR line per source and minute. Only a retryable failure (a timeout, an unavailable vendor) and a thrown push follow onPushError.

Emulator​

The pull subscriber honors PUBSUB_EMULATOR_HOST. For explicit configuration, set settings.apiEndpoint (e.g. localhost:8085).

gcloud beta emulators pubsub start --host-port=localhost:8085
export PUBSUB_EMULATOR_HOST=localhost:8085

Lifecycle​

The pull source is a long-running listener. The other server sources (express, lambda, cloudfunction, fetch) are request handlers; pull works like this:

  1. init() returns the Source.Instance. When the collector runs, the source opens the streaming subscriber, which runs in the background.
  2. The instance's push() called without arguments does nothing; called with a synthetic message it runs the same handler (tests, simulation).
  3. destroy() closes the subscriber. On shutdown the source stops accepting new messages, drains in-flight ack/nacks, and force-closes after shutdownTimeoutMs.

Troubleshooting​

  • NOT_FOUND on the subscription: the subscription does not exist. Create it via gcloud pubsub subscriptions create (walkeros setup cannot load this source today, see Setup).
  • PERMISSION_DENIED / UNAUTHENTICATED on the pull subscriber: the runtime service account lacks roles/pubsub.subscriber on the subscription.
  • Push handler returns 401 for every message: OIDC misconfiguration. Verify settings.audience matches your endpoint URL exactly, including scheme and path.
  • Messages decoded incorrectly: JSON is the default. For plain-text or binary payloads, switch the decoder to text or raw and add a source.before transformer that turns the payload into an event.

Next steps​

💡 Need implementation support?
elbwalker offers hands-on support: setup review, measurement planning, destination mapping, and live troubleshooting. Book a 2-hour session (€399)