Understanding walkerOS Destinations

Overview

Destinations receive processed events from the collector and deliver them to third-party tools (analytics, marketing, data warehouses).

Core principle: Destinations transform and deliver. They don't capture or process—that's sources and collector.

Destination Interface

See packages/core/src/types/destination.ts for canonical interface.

Method	Purpose	Required
init(context)	Load scripts, authenticate	Optional
push(event, context)	Transform and send event	Required
pushBatch(batch, context)	Batch processing	Optional
destroy(context)	Cleanup on shutdown	Optional
config	Settings, mapping, consent	Required

destroy?: DestroyFn — Optional cleanup method. Called during command('shutdown'). Use to close DB connections, flush buffers, or release SDK clients. Receives { id, config, env, logger }.

The env Pattern

Destinations use dependency injection via env for external APIs. This enables testing without mocking.

// Destination defines its env type
export interface Env extends DestinationWeb.Env {
  window: {
    gtag: Gtag.Gtag;
    dataLayer: unknown[];
  };
}

// Destination uses env, not globals
async function push(event, context) {
  const { env } = context;
  env.window.gtag('event', mappedName, mappedData);
}

Testing with env

REQUIRED SKILL: See testing-strategy for full testing patterns.

import { mockEnv } from '@walkeros/core';
import { examples } from '../dev';

const calls: Array<{ path: string[]; args: unknown[] }> = [];
const testEnv = mockEnv(examples.env.push, (path, args) => {
  calls.push({ path, args });
});

await destination.push(event, { ...context, env: testEnv });

expect(calls).toContainEqual({
  path: ['window', 'gtag'],
  args: ['event', 'purchase', expect.any(Object)],
});

The env as observability seam

The env pattern is also the interception seam for trace-level vendor-call capture. At trace level the collector may attach an observe recorder under the observe key of the env, and @walkeros/web-core's getEnv wraps the callables the destination declared as observable, recording each call. A destination that reaches its vendor through getEnv(env) is automatically observable; one that reaches a global directly, bypassing env, is not. This is one more reason to route every external call through env.

Rules that keep the seam intact:

• init keeps an injected env. A client built in init (SDK client, auth client, writer) is taken from env when present and created only otherwise (env?.BigQuery ?? BigQuery). Every key the mock env declares must be read as env.<key> in src.
• simulation names the request call (sendServer, call:PubSub.topic.publishMessage, call:JSONWriter.appendRows), not a constructor. The one recorder is observeEnv in @walkeros/core; it navigates constructors, factories and Promises, and records only the leaf.
• Trace-mode limit. CLI and MCP simulate inject the mock env before init, so a client built in init is recorded. Runtime trace mode wraps the env per push only, so calls through a client built in init are not recorded there.
• Client IP and user agent. Server conversion-API destinations (Meta, TikTok, Snapchat, Pinterest, Bing, Reddit, Criteo, X, Data Manager, Piwik PRO) expose ip and userAgent settings (Mapping.Value | false), defaulting to ['ingest.ip', 'event.user.ip'] and ['ingest.userAgent', 'event.user.userAgent'], resolved with getMappingValue(createMappingRoot(ingest, event), setting, { collector }). false disables, an explicit mapped vendor field wins, each package carries its own few lines (no shared helper).

Destination Config

config: {
  settings: { /* destination-specific */ },
  mapping: { /* event transformation rules */ },
  data: { /* global data mapping */ },
  consent: { /* required consent states */ },
  policy: { /* processing rules */ },
  queue: boolean,  // queue events
  dryRun: boolean, // test mode
  queueMax: number,  // consent-queued events cap (default 1000)
  dlqMax: number,    // dead-letter queue cap (default 100)
}

Buffer bounds

Each destination keeps two internal buffers: queuePush (consent-denied events) and dlq (failed pushes). Both are size-bounded with FIFO drop-oldest eviction. Defaults: queueMax: 1000, dlqMax: 100. Set either on a destination's config to override per destination. Operators read drop counts from collector.status.dropped[stepId('destination', id)]?.queue (consent-denied evictions) and ?.dlq (DLQ evictions). Build the key with stepId() from @walkeros/core. Point-in-time sizes stay on collector.status.destinations[id].queuePushSize / dlqSize.

queuePush drains back into the destination once consent allows delivery. The DLQ is never replayed: an event that reaches it is gone once it is evicted, which is what makes the delivery rules below worth getting right.

Delivery timeout

config.timeout bounds one delivery. A push or pushBatch that has not settled within it is abandoned, its events are routed to the DLQ and failed is incremented. The default is 10000 ms, applied whenever timeout is 0 or unset, so one slow destination never wedges the collector push.

A destination's own client timeout has to fit inside it, retry delays included. Many SDK defaults are longer (30 s in several HTTP clients), and a longer one produces the worst outcome available: the collector stops waiting at 10 s, dead-letters the batch and counts it lost, then the request lands anyway a few seconds later. Those events are both counted lost and actually written. Derive the client's per-attempt timeout from config.timeout minus what the retry delays can take, split across the attempts, so the last attempt finishes before the race does.

The same bound applies at shutdown, where the collector races each pending batch flush against a fixed 5000 ms step timeout before destroying the destination. A destination whose retries need longer than that loses them on shutdown.

Batch scheduling

Set config.batch on a destination (with pushBatch implemented) to batch all of its events into one shared buffer. No '* *' wildcard mapping rule is needed. The configuration shape is batch?: number | { wait?, size?, age? } at both the destination-config layer and the mapping-rule layer.

pushBatch is the gate, and a missing one fails silently. The collector engages batching only when destination.pushBatch exists. Set config.batch on a destination that lacks it and every event is sent one at a time, with no warning and no error: the config is accepted and inert. When adding batching to a destination, implementing pushBatch is the change; config.batch alone buys nothing.

• wait (ms): debounce window. The timer resets on every push. Legacy form batch: 1000 is shorthand for { wait: 1000 }.
• size: hard count cap. Default 1000. Flushes immediately when reached.
• age (ms): hard age cap since the first entry of the current window. Default 30000. Prevents debounce starvation under sustained load.

A mapping rule's own batch splits that entity-action into its own buffer and overrides config.batch per field (rule ?? config ?? default). To batch only specific events, omit config.batch and set batch on those rules. Pending batches flush on shutdown.

Per-event metadata

Batch.entries[] carries per-event { event, ingest?, respond?, rule?, data? }. Destinations that need per-event request IDs or HTTP responses (BigQuery, mParticle, HubSpot) should read entries instead of assuming all events in a batch share one ingest. batch.events and batch.data are derived views kept for backward compatibility.

Failure handling

pushBatch has three return outcomes:

• Resolves void - the whole batch succeeded; every entry counts as delivered.
• Throws or rejects - the whole batch failed. Every entry is routed to the destination's dlq and status.destinations[id].failed is incremented by the batch size.
• Resolves a BatchOutcome - partial failure. { failed: [{ index, error? }] } names the entries that did not make it, by index into batch.entries. Only those are DLQ'd; the rest count as delivered. Use this when the vendor reports per-row results (BigQuery's rowErrors is the reference case), so a later DLQ retry does not rewrite rows that already landed.

A BatchOutcome is for sinks that report per row, and for nothing else. A sink that succeeds or fails as a whole (one SQL INSERT, one file write, one publish) must never report an all-failed outcome in place of throwing. Only a throw records a transport failure on the breaker, and only a flush with at least one delivered entry records a success, so an outcome where nothing succeeded records neither: a total outage passes the breaker unseen and it never opens. For a whole-batch sink there are two answers and no third one, resolve void or throw.

Batching, the delivery timeout, the DLQ and the breaker are the collector's; per-call retry is the vendor SDK's (BigQuery, Kafka, HubSpot and the AWS SDK each have their own backoff semantics). Where a destination relies on that SDK retry, it keeps the attempts and their delays inside config.timeout, and makes a repeat safe through an idempotency key or a deduplication token computed once and reused across attempts. Without that, a retry of a delivery the vendor had already accepted duplicates data. Counters (count, out) are bumped only after a successful flush.

Three rules every destination follows. First, no retry loops, backoff or DLQ code of its own: each request is sent once through the vendor SDK, whose built-in retry stays at its defaults, because a standard retry belongs in the collector, not in each package. Second, init is offline: it validates the config and builds clients, but makes no network call, loads no credentials and creates no remote resources; authentication and lookups happen on the first delivery, and provisioning belongs to setup. Third, every failure, thrown or reported in a BatchOutcome, is an Error that names the cause (and, for a configuration error, the fix) and carries code (the vendor error name or a package code), status (the HTTP status when the vendor answered) and retryable. Take retryable from the vendor SDK's own public classifier where one exists, else from documented codes or statuses, and keep the SDK error reachable as cause. @walkeros/server-destination-aws is the reference implementation.

Operators also see status.destinations[id].inFlightBatch: the number of events buffered but not yet delivered.

Out-of-band errors from an EventEmitter SDK

tryCatchAsync only catches errors on the awaited push/pushBatch path. If a destination owns an SDK object that is an EventEmitter (a BigQuery StreamConnection, a Redis client, a Kafka producer), that object can emit 'error' on a detached tick with no awaiter. An EventEmitter that emits 'error' with zero listeners throws synchronously and crashes the process.

A destination that owns such an object MUST attach an 'error' listener and route the error through context.reportError, which is available on every step context:

• context.reportError(err) (no event): a connection-level error between pushes. Logs (redacted) and bumps status.connectionErrors[stepId]. Does not count as a failed event.
• context.reportError(err, event): a specific event was lost. Routes it to the DLQ and bumps failed, exactly like an in-band push failure.

reportError is guarded and must never be called in a way that throws back into the emitter tick. The Pub/Sub pull source (sources/gcp/src/pubsub/pull) is the reference for attaching an 'error' listener. The runner's process guards are a backstop, but the listener is what gives clean DLQ routing, attribution, and redaction. Optionally pair this with the breaker config so a persistently broken transport stops retrying every event.

Require vs Consent

Two separate mechanisms control when destinations receive events:

Mechanism	Purpose	Scope	Effect
require	Delay initialization	Whole destination	Destination stays in pending until all required events fire (e.g., walker consent)
consent	Filter events	Per-event	Events without matching consent are silently skipped or queued

Require gates the destination lifecycle. A destination with require: ["consent"] does not exist in the collector until consent state is present: a "walker consent" event, or starting consent (startFlow's consent, which walkeros push --simulate sets with --consent). Until then, events are queued internally.

Consent gates individual event delivery. A destination with consent: { marketing: true } only receives events where the collector's consent state (or the event's own consent field) includes { marketing: true }.

State refresh on flush: When queued events are flushed to a destination, they receive the current collector state (consent, user, globals) — not the stale state from when they were originally captured. Any state-mutation command (walker consent, walker user, walker globals, etc.) triggers a flush attempt. The consent gate still applies: events without required consent simply return to the queue.

Both can be combined:

{
  "config": {
    "require": ["consent"],
    "consent": { "marketing": true }
  }
}

This means: don't initialize until consent fires, then only accept events with marketing consent. The state that satisfies require reaches on() before the destination's queued events are pushed.

Simulation impact: require causes "destination not found" errors in flow_simulate because the destination stays pending. Remove require temporarily for simulation testing.

Policy

Policy modifies the event BEFORE mapping rules run. Defined at config level (applies to all events) or rule level (applies to specific events):

{
  "config": {
    "policy": {
      "user_data.email": {
        "key": "user.email",
        "consent": { "marketing": true }
      }
    }
  }
}

Policy supports consent-gated field injection — fields are only added when the required consent is present in the event.

Destination Paths

Type	Path	Examples
Web	packages/web/destinations/	gtag, meta, api, piwikpro, plausible
Server	packages/server/destinations/	aws, gcp, meta

Template Destination

Use as starting point: packages/web/destinations/plausible/

Transformer Wiring

Destinations can wire to their own transformer chain via the before property. It runs for this destination only, after the collector chain (collector.next, shared by every destination); a stop in it skips only this destination:

destinations: {
  gtag: {
    code: destinationGtag,
    before: 'redact'  // First transformer to run before this destination
  }
}

The transformer chain runs after collector enrichment, before the destination receives events. Each destination can have its own chain. See understanding-transformers for chain details.

Consent: Two-Layer Pattern

Destinations that integrate vendor SDKs typically need two consent layers:

Layer 1: config.consent — gates walkerOS event delivery. If consent is not granted, events don't reach the destination. This is the primary barrier.

Layer 2: on('consent') — controls vendor SDK internals. Even when walkerOS stops sending events, the vendor SDK may still run its own behaviors (DOM capture, polling, fetching configs). Use on('consent') to pause/resume these.

on(type, context) {
  if (type !== 'consent') return;
  const consent = context.data;
  // Derive from config.consent keys — don't hardcode consent names
  const granted = Object.keys(config.consent || {}).every(k => consent[k]);
  vendorSdk.setOptOut(!granted);
}

Both layers are needed for complete consent compliance. config.consent prevents data flow. on('consent') prevents vendor SDK side effects.

on('consent') runs before the first push on every activation path (require, walker destination at runtime, state passed to walker run, state set before run) and is delivered exactly once per state change. It may be async: the collector awaits it, bounded by config.timeout. A handler that does not settle in that time, or rejects, holds that destination: its events stay queued until a later delivery of the state settles, and are discarded at the next walker run. Lifecycle handlers such as on('run') are awaited too, bounded by the same timeout, and never hold events. An async state handler must settle: until it settles or reaches config.timeout, that destination receives no events. elb commands and startFlow resolve only after the destinations' on() handlers for that command have settled or timed out.

Response Delegation (env.respond)

Destinations can customize HTTP responses by calling context.env.respond?.({ body, status?, headers? }). This is useful for destinations that need to signal success/failure back to the HTTP caller. First call wins (idempotent). The respond function is optional — only present when the source provides one.

Setup (optional)

Destinations can implement an optional setup() lifecycle to provision external resources, for example a BigQuery dataset and table, a Pub/Sub topic, or a warehouse schema. Setup is never invoked by the runtime, push, init, or deploy. It runs only when an operator explicitly types walkeros setup destination.<name>.

The signature is (ctx: LifecycleContext<Config<T>, Env<T>>) => Promise<unknown>, where LifecycleContext carries { id, config, env, logger }. Idempotency is the package's responsibility: the framework adds no opinion. Use resolveSetup(ctx.config.setup, DEFAULTS) from @walkeros/core to normalize the boolean | object shape into a concrete options object.

See walkeros-create-destination, walkeros-understanding-sources, walkeros-understanding-stores, and the walkeros setup CLI documentation for the authoring template and operator workflow.

Related Skills

• walkeros-understanding-flow - How destinations fit in architecture
• walkeros-understanding-mapping - Configure transformations
• walkeros-create-destination - Create new destination

Source Files:

• packages/core/src/types/destination.ts - Interface

Package READMEs:

• packages/web/destinations/gtag/README.md - gtag example
• packages/web/destinations/plausible/README.md - Plausible (template)

Documentation:

• Website: Destinations - Overview
• Website: Create Your Own - Guide