Google Pub/Sub
Subscribes to a Google Cloud Pub/Sub topic and forwards each delivered message to the walkerOS collector. The package ships two delivery models: a long-running streaming pull subscriber for container deployments, and an HTTP push webhook handler for serverless. It has decoders for JSON, text, and raw payloads, idempotent subscription provisioning with optional dead-letter wiring, and optional OIDC verification on the push handler.
The source ships inside @walkeros/server-source-gcp alongside the Cloud Functions handler; install the package once and import sourcePubSubPull or sourcePubSubPush.
Pub/Sub is a server source in the walkerOS flow:
Forwards each Pub/Sub message to the collector. Pull mode runs a streaming subscriber in long-lived containers; push mode receives HTTP envelopes in serverless deployments.
Installation
npm install @walkeros/server-source-gcpChoosing pull vs push
| Aspect | Pull (sourcePubSubPull) | Push (sourcePubSubPush) |
|---|---|---|
| Process model | Long-running streaming subscriber | Stateless HTTP request handler |
| Hosting | Cloud Run with min-instances >= 1, GKE, GCE, on-prem | Cloud Run scale-to-zero, Cloud Functions, Lambda |
| Idle cost | Pay for always-on compute | Zero when idle |
| Throughput | Highest (gRPC streaming) | Limited by HTTP overhead |
| Latency | Lowest (warm stream, single-digit ms) | Higher (HTTP per message, cold-start adds 100-500ms) |
| Auth direction | Subscriber holds credentials, connects outbound | Pub/Sub holds credentials, posts inbound (OIDC verify) |
| Subscription type | subscription.type = "pull" | subscription.type = "push" with pushConfig.pushEndpoint |
| Best fit | Long-running collector services, high-volume streams | Serverless walkerOS endpoints, hybrid setups |
Both modes share the @walkeros/server-source-gcp package and the decoders; only pull has a setup command and SDK authentication. Pick the one that matches your hosting model.
Pull subscriber
import { sourcePubSubPull } from '@walkeros/server-source-gcp';
import { startFlow } from '@walkeros/collector';
await startFlow({
sources: {
pubsub: {
code: sourcePubSubPull,
config: {
settings: {
projectId: 'YOUR_PROJECT_ID',
subscription: 'YOUR_SUBSCRIPTION_NAME',
},
},
},
},
destinations: {
// your destinations
},
});The pull source is event-driven: consumption starts when the collector runs, and the source then opens the streaming subscription and forwards each delivered message to the collector via env.push. The source's push() called without arguments does nothing; called with a synthetic message it runs the same handler (tests, simulation). destroy() closes the subscriber gracefully, honoring shutdownTimeoutMs (default 30000).
Pull settings
| Field | Type | Default | Description |
|---|---|---|---|
projectId | string | (required) | GCP project id. |
subscription | string | (required) | Subscription short name. |
topic | string | (optional) | Topic short name. Setup needs it to create the subscription (and setup.createTopic to create the topic). |
credentials | string | ServiceAccountCredentials | (ADC fallback) | Service account JSON (parsed object or JSON string). Set on config.credentials (sibling of settings). |
apiEndpoint | string | (SDK default) | Override for the emulator (e.g. localhost:8085). |
decoder | 'json' | 'text' | 'raw' | 'json' | How the subscriber decodes message bodies before forwarding. |
flowControl | { maxMessages?, maxBytes? } | { maxMessages: 100, maxBytes: 10 MB } | Subscriber pull rate (in-flight cap). |
ackDeadline | number | 60 | Per-message ack window in seconds. |
shutdownTimeoutMs | number | 30000 | Graceful drain budget on destroy(). |
onPushError | 'nack' | 'ack' | 'nack' | Behavior when forwarding to the collector throws or fails for a retryable reason. |
Push webhook
import express from 'express';
import { Source } from '@walkeros/core';
import {
sourcePubSubPush,
type SourcePubSubPush,
} from '@walkeros/server-source-gcp';
import { startFlow } from '@walkeros/collector';
const { collector } = await startFlow({
sources: {
pubsub: {
code: sourcePubSubPush,
config: { settings: { decoder: 'json' } },
},
},
destinations: {
// your destinations
},
});
const app = express();
app.use(express.json());
app.post(
'/pubsub-push',
Source.getSource<SourcePubSubPush.Types>(collector, 'pubsub').push,
);
app.listen(8080);Pub/Sub posts each message envelope to the configured endpoint as a JSON body matching the Pub/Sub push spec. The handler decodes the envelope, base64-decodes the data field, runs the configured decoder, and forwards to the collector. Status codes:
200: forwarded successfully, or failed for a reason a redelivery cannot fix (it is in the destination's dead-letter queue and in the collector's ERROR line).200withdropped: true: the payload decoded tonull; nothing is forwarded and nothing is logged.400: the message is permanently invalid, either a malformed envelope (missingmessage.data, invalid base64) or an event the pipeline rejected. A retry cannot fix either, so the answer is a nack.401: OIDC verification failed (when enabled).500: collector push failed on an otherwise valid message for a retryable reason, or threw; Pub/Sub will retry per the subscription's retry policy.
Because a permanently invalid message is nacked rather than acknowledged, Pub/Sub keeps redelivering it. Set a dead-letter policy on the subscription: that policy, not the source, is the poison-message guard.
Push settings
Push settings: projectId (optional, not used by the handler), decoder, and:
| Field | Type | Default | Description |
|---|---|---|---|
verifyOidc | boolean | false | When true, verifies the bearer token Pub/Sub attaches to push requests. |
audience | string | - | Required when verifyOidc is true. Must match your endpoint URL exactly. |
Authentication
Three modes for the pull subscriber, evaluated in order:
- Application Default Credentials (ADC). Default. Works on GCP-native runtimes (Cloud Run, Cloud Functions, GKE) and locally with
gcloud auth application-default login. - Service account JSON. Pass
config.credentials, either parsed or as JSON string (the source JSON-parses strings). Combine with$env.NAMEto inject from an environment variable. - Pre-configured client. Pass an existing
PubSubSDK instance assettings.client.
For the push handler, auth is reversed: Pub/Sub holds the identity and signs requests with an OIDC token. The handler verifies the token against GCP public keys when verifyOidc: true and audience is set. Off by default because a misconfigured OIDC answers every message with 401, so opt in once your endpoint is publicly reachable.
Setup
In a flow file the step names the pull export with "import", because the package has no default export, and the topic the subscription is created on:
"sources": {
"pubsub": {
"package": "@walkeros/server-source-gcp",
"import": "sourcePubSubPull",
"config": {
"settings": {
"projectId": "YOUR_PROJECT_ID",
"subscription": "YOUR_SUBSCRIPTION_NAME",
"topic": "YOUR_TOPIC_NAME"
},
"setup": true
}
}
}walkeros setup source.pubsub would provision the subscription once per environment; today it cannot load the source:
walkeros setup source.pubsub stops with Package @walkeros/server-source-gcp has no export "sourcePubSubPull" (or has no default export without "import") and creates nothing. Create the subscription with gcloud pubsub subscriptions create or your infrastructure code. The rest of this section describes the source's own setup function.
Idempotent. Defaults: Pub/Sub's own (10-second ack deadline), project-default retention, no filter, no dead-letter policy. Drift on ackDeadlineSeconds, messageRetentionDuration, deadLetterPolicy, enableMessageOrdering, or filter logs one setup.drift warning per field ({ field, declared, actual }) on the logger the setup function receives, and never auto-mutates.
config.setup:
- omitted or
false(default): no provisioning. true: provisions the subscription onsettings.topic(an existing topic) with safe defaults. Withoutsettings.topicsetup creates nothing, andwalkeros setupstill reportssetup: ok.{ ackDeadlineSeconds, messageRetentionDuration, filter, deadLetterPolicy, retryPolicy, ... }: object form for explicit overrides.
Optional one-shot helpers:
setup.createTopic: true: auto-create the topic if it does not exist (requiressettings.topic). Useful when source and topic owners share a deployment.setup.deadLetterPolicy.createDeadLetterTopic: true: auto-create the dead-letter topic referenced indeadLetterPolicy.deadLetterTopic.
The provisioning identity needs pubsub.subscriptions.create (and pubsub.topics.create if auto-creating). The runtime pull identity needs roles/pubsub.subscriber on the subscription.
Topic provisioning is owned by the Pub/Sub destination when the topic is the producer's responsibility; the source can also auto-create as a convenience for self-contained setups.
Decoders
| Decoder | Behavior |
|---|---|
json (default) | JSON.parse(data.toString('utf8')) |
text | The UTF-8 string, forwarded as data.payload (pull) or as the push value (push) |
raw | The bytes, as a base64 string in data.payload (pull) or the Buffer (push) |
Neither text nor raw carries an event name: without a source.before transformer that builds the event, the collector rejects it as invalid, so pull acknowledges and drops the message and push answers 400. A JSON payload of null is acknowledged and dropped with a DEBUG line only (push: 200 with dropped: true, no log line).
A decoder that throws on malformed payloads triggers onPushError (pull mode): nack redelivers, ack drops it; the decode failure is an ERROR line. In pull mode a failure a redelivery cannot fix (a missing setting, a 4xx from the vendor, a destination that is not ready) and invalid input are acknowledged: the failure is in the destination's dead-letter queue and in one ERROR line from the collector, and invalid input is counted as rejected with one ERROR line per source and minute. Only a retryable failure (a timeout, an unavailable vendor) and a thrown push follow onPushError.
Emulator
The pull subscriber honors PUBSUB_EMULATOR_HOST. For explicit configuration, set settings.apiEndpoint (e.g. localhost:8085).
gcloud beta emulators pubsub start --host-port=localhost:8085
export PUBSUB_EMULATOR_HOST=localhost:8085Lifecycle
The pull source is a long-running listener. The other server sources (express, lambda, cloudfunction, fetch) are request handlers; pull works like this:
init()returns theSource.Instance. When the collector runs, the source opens the streaming subscriber, which runs in the background.- The instance's
push()called without arguments does nothing; called with a synthetic message it runs the same handler (tests, simulation). destroy()closes the subscriber. On shutdown the source stops accepting new messages, drains in-flight ack/nacks, and force-closes aftershutdownTimeoutMs.
Troubleshooting
NOT_FOUNDon the subscription: the subscription does not exist. Create it viagcloud pubsub subscriptions create(walkeros setupcannot load this source today, see Setup).PERMISSION_DENIED/UNAUTHENTICATEDon the pull subscriber: the runtime service account lacksroles/pubsub.subscriberon the subscription.- Push handler returns 401 for every message: OIDC misconfiguration. Verify
settings.audiencematches your endpoint URL exactly, including scheme and path. - Messages decoded incorrectly: JSON is the default. For plain-text or binary payloads, switch the decoder to
textorrawand add asource.beforetransformer that turns the payload into an event.
Next steps
- Pub/Sub destination for publishing events to a Pub/Sub topic
- GCP source overview for the Cloud Functions HTTP handler