> Part of the walkerOS documentation. Project overview and full index: # Understanding walkerOS Flow ## Overview[​](#overview "Direct link to Overview") walkerOS follows a **Source → Collector → Destination(s)** architecture for composable, modular event processing. **Core principle:** Separation of concerns. Each component has one job. Components are composable and replaceable. ## The Flow Pattern[​](#the-flow-pattern "Direct link to The Flow Pattern") ``` [Source.before] → Sources → [Source.next] → Collector → [Dest.before] → Destinations → [Dest.next] (Preprocessing) (Capture) (Pre-chain) (Processing) (Post-chain) (Delivery) (Post-push) Consent-exempt: Post-consent: - Decode - Validation - Event creation - Validation - Push - Audit logging - Validate format - Enrichment - Enrichment - Enrichment - Send - Response parsing - Authenticate - Redaction - Consent check - Routing - Store - Webhooks ``` ## Key Concepts[​](#key-concepts "Direct link to Key Concepts") ### Composability[​](#composability "Direct link to Composability") A Flow combines components. You can: * Use multiple sources feeding one collector * Route events to multiple destinations * Swap components without changing others ### The Flow Type[​](#the-flow-type "Direct link to The Flow Type") See [packages/core/src/types/flow.ts](https://github.com/elbwalker/walkerOS/blob/main/packages/core/src/types/flow.ts) for the canonical interface. ``` // Conceptual structure (see source for full type) interface Flow { stores?: Record; sources?: Record; transformers?: Record; destinations?: Record; collector?: Collector.InitConfig; } ``` ### Universal Push Interface[​](#universal-push-interface "Direct link to Universal Push Interface") **All components communicate via `push` functions:** | Component | Push Signature | Purpose | | ----------- | ----------------------------- | --------------------- | | Source | `push(input) → events` | Capture external data | | Collector | `push(event) → void` | Process and route | | Destination | `push(event, context) → void` | Transform and deliver | The `elb()` function is an alias for `collector.push` - used for component wiring. ### startFlow Helper[​](#startflow-helper "Direct link to startFlow Helper") See [packages/collector/src/flow.ts](https://github.com/elbwalker/walkerOS/blob/main/packages/collector/src/flow.ts) for the `startFlow` function. ``` import { startFlow } from '@walkeros/collector'; const { collector, elb } = await startFlow({ stores: { /* key-value storage, init first, destroy last */ }, sources: { /* ... */ }, transformers: { /* ... */ }, destinations: { /* ... */ }, }); ``` ## Ingest: Mutable Pipeline Context[​](#ingest-mutable-pipeline-context "Direct link to Ingest: Mutable Pipeline Context") walkerOS uses a two-layer data model: * **Event** - strict schema, structured analytics data (name, data, context, etc.) * **Ingest** - free-form mutable context that flows alongside events Any step can read and write arbitrary keys on ingest. The runtime manages `_meta`: * `_meta.hops` - increments per step (safety valve at 256) * `_meta.path` - ordered list of step IDs visited (path\[0] = source ID) ``` // In a transformer push: async (event, context) => { context.ingest.botScore = 0.95; // Write freely context.ingest.geo = { country: 'DE' }; console.log(context.ingest._meta.path); // ['express', 'validate', 'enrich'] return { event }; }; ``` Ingest is cloned per destination to prevent cross-contamination in parallel processing. After a destination push, the response is available at `ingest._response`. ## Stores[​](#stores "Direct link to Stores") Three pipeline components (Source / Transformer / Destination) plus Stores as **passive infrastructure**. Stores are not a fourth pipeline stage - they're key-value storage that other components consume via `env`. They have no `push`, no `next`, no `before`; they sit alongside the pipeline rather than inside it. * Referenced via `$store.storeId` in `env` values (bundled mode) or passed directly as store instances (integrated mode) * **Init first, destroy last** - stores are available before any source, transformer, or destination starts, and outlive them on shutdown * **No chains** - stores don't participate in the event pipeline. Components access them through their `env`. * Implementations: `@walkeros/server-store-fs` (async, filesystem), `@walkeros/server-store-s3` (async, S3-compatible), `@walkeros/server-store-gcs`, `@walkeros/server-store-sheets`. The collector ships a built-in in-memory cache tier — enable it on any store via `Flow.Store.cache` instead of declaring a separate memory store. ``` { "stores": { "data": { "package": "@walkeros/server-store-fs" } }, "transformers": { "fingerprint": { "package": "@walkeros/server-transformer-fingerprint", "env": { "store": "$store.data" } } } } ``` See [walkeros-understanding-stores](https://www.walkeros.io/skills/walkeros-understanding-stores.md) for the full store interface and lifecycle. ## Separation of Concerns[​](#separation-of-concerns "Direct link to Separation of Concerns") | Concern | Handled By | NOT Handled By | | ---------------- | -------------- | ----------------------- | | Event capture | Sources | Collector, Destinations | | Event structure | Event model | Components | | Consent checking | Collector | Sources, Destinations | | Transformation | Mapping system | Raw push calls | | Delivery | Destinations | Sources, Collector | ## Step-Level Primitives[​](#step-level-primitives "Direct link to Step-Level Primitives") Every step (source, transformer, destination) supports a small set of inline primitives alongside its package wiring: `cache`, `mapping`, and `consent`. Event shapes are not a step-level primitive. They live in the top-level `contract` block (a sibling of `flows`) as named JSON Schemas, and enforcement is an explicit `@walkeros/transformer-validate` step that references a contract via `$contract.`. See [Website: Contract](https://www.walkeros.io/docs/getting-started/flow/contract.md) for the contract shape and [Website: Validate](https://www.walkeros.io/docs/getting-started/flow/validate.md) for runtime enforcement. ### How a step references its implementation[​](#how-a-step-references-its-implementation "Direct link to How a step references its implementation") Every step picks exactly one of three forms to point at its implementation: * `package: ""` alone loads the package's default export. The common case for sources and destinations. * `package: ""` plus `import: ""` loads a specific named export from that package. Use when a package ships multiple named exports or has no default export. * `code: { push, type?, init? }` is inline implementation, no package wiring. Useful for one-off custom logic in TypeScript flows. There is no string form of `code`. A bare step with no `package`, `import`, or `code` is a valid no-op for all four step kinds (handy as a chain-only or mapping-only transformer). ## Transformer Chains[​](#transformer-chains "Direct link to Transformer Chains") Transformers run at two points in the pipeline, configured via `next` and `before`: ### Pre-Collector Chain[​](#pre-collector-chain "Direct link to Pre-Collector Chain") Runs after source captures event, before collector processing: **Bundled mode (flow\.json):** ``` { "sources": { "browser": { "package": "@walkeros/web-source-browser", "next": "enrich" } }, "transformers": { "enrich": { "package": "@walkeros/transformer-enricher", "next": "redact" }, "redact": { "package": "@walkeros/transformer-redact" } } } ``` **Integrated mode (TypeScript):** ``` sources: { browser: { code: sourceBrowser, next: 'enrich' } }, transformers: { enrich: { code: transformerEnrich, config: { next: 'redact' } }, redact: { code: transformerRedact } } ``` Note: In flow\.json, `next` is at the reference level. The CLI bundler automatically moves it into `config.next` for runtime - you don't need to handle this yourself. ### Transformer before chain[​](#transformer-before-chain "Direct link to Transformer before chain") Each transformer can have its own `before` chain that runs before its push function: ``` { "transformers": { "enrich": { "before": "lookup", "package": "@walkeros/transformer-enricher" }, "lookup": { "package": "@walkeros/transformer-lookup" } } } ``` ### Post-Collector Chain[​](#post-collector-chain "Direct link to Post-Collector Chain") Runs after collector enrichment, before destination receives event: **Bundled mode (flow\.json):** ``` { "destinations": { "gtag": { "package": "@walkeros/web-destination-gtag", "before": "redact" } }, "transformers": { "redact": { "package": "@walkeros/transformer-redact" } } } ``` **Integrated mode (TypeScript):** ``` destinations: { gtag: { code: destinationGtag, before: 'redact' } }, transformers: { redact: { code: transformerRedact } } ``` ### Post-push chain (`destination.next`)[​](#post-push-chain-destinationnext "Direct link to post-push-chain-destinationnext") Runs after destination push completes. The push response is available at `context.ingest._response`: **Bundled mode (flow\.json):** ``` { "destinations": { "api": { "package": "@walkeros/server-destination-api", "next": "auditLog" } }, "transformers": { "auditLog": { "package": "@walkeros/transformer-audit" } } } ``` **Integrated mode (TypeScript):** ``` destinations: { api: { code: destinationApi, next: 'auditLog' } }, transformers: { auditLog: { code: transformerAudit } } ``` ### Chain Resolution[​](#chain-resolution "Direct link to Chain Resolution") * `source.before` → consent-exempt preprocessing chain * `source.next` → starts pre-collector chain * `transformer.before` → pre-transform enrichment chain * `transformer.next` (flow\.json) or `transformer.config.next` (runtime) → links transformers * `destination.before` → starts post-collector chain per destination * `destination.next` → post-push processing chain ## Cross-Flow References (`$flow`)[​](#cross-flow-references-flow "Direct link to cross-flow-references-flow") When a single flow\.json defines multiple flows, any flow can pull values from another flow's `config` block via `$flow.(.)?`. The most common case is wiring a web flow's API destination to a server flow's deployed URL, so the two stay in sync without duplication. ``` { "version": 4, "flows": { "server": { "config": { "platform": "server", "url": "https://collect.example.com" }, "sources": { "http": { "package": "@walkeros/server-source-express" } } }, "web": { "config": { "platform": "web" }, "destinations": { "api": { "package": "@walkeros/web-destination-api", "config": { "settings": { "url": "$flow.server.url" } } } } } } } ``` `validate` warns on unresolved `$flow` references (lenient), `bundle` errors out (strict), so production builds never ship with an empty cross-flow value. ## Step Examples[​](#step-examples "Direct link to Step Examples") Each step in a flow (source, transformer, destination) can ship **step examples** -- structured `{ in, out }` pairs that define expected input/output behavior. ### The Three Type Zones[​](#the-three-type-zones "Direct link to The Three Type Zones") Steps sit at boundaries between arbitrary formats and walkerOS events: * **Source:** arbitrary `in` (HTTP request, DOM event) → walkerOS event `out` * **Transformer:** walkerOS event `in` → walkerOS event `out` (or `false`) * **Destination:** walkerOS event `in` → arbitrary `out` (vendor API call) See [using-step-examples](https://www.walkeros.io/skills/walkeros-using-step-examples.md) for the full ASCII diagram and detailed explanation. ### Example: Step with Examples in Flow Config[​](#example-step-with-examples-in-flow-config "Direct link to Example: Step with Examples in Flow Config") ``` { "destinations": { "gtag": { "package": "@walkeros/web-destination-gtag", "config": { "measurementId": "G-XXXXXX" }, "examples": { "purchase": { "in": { "name": "order complete", "data": { "id": "ORD-123", "total": 149.97 } }, "out": [ "event", "purchase", { "transaction_id": "ORD-123", "value": 149.97 } ] } } } } } ``` Step examples enable `it.each` testing, CLI simulation with `--example`, and deep validation with `--deep`. See [using-step-examples](https://www.walkeros.io/skills/walkeros-using-step-examples.md) for the complete lifecycle. ## Flow Graph Connection Rules[​](#flow-graph-connection-rules "Direct link to Flow Graph Connection Rules") This section defines which components can connect to which, and how chains are resolved at runtime. Use it as the canonical reference for building flow graphs, validating configurations, and rendering UI visualizations. ### Valid connection matrix[​](#valid-connection-matrix "Direct link to Valid connection matrix") | From | To | Via Field | Valid? | | ----------- | ----------- | --------------------------- | --------------------- | | Source | Transformer | `source.before` | Yes (consent-exempt) | | Source | Transformer | `source.next` | Yes (pre-collector) | | Source | Collector | (implicit, no next) | Yes | | Source | Source | - | No | | Source | Destination | - | No | | Transformer | Transformer | `transformer.before` | Yes (pre-transform) | | Transformer | Transformer | `transformer.next` | Yes (chain continues) | | Transformer | Collector | (implicit, pre-chain ends) | Yes | | Transformer | Destination | (implicit, post-chain ends) | Yes | | Collector | Destination | (implicit, no before) | Yes | | Collector | Transformer | `destination.before` | Yes (post-chain) | | Destination | Transformer | `destination.next` | Yes (post-push) | | Collector | Source | - | No | ### Pre-transformer chains (`source.next`)[​](#pre-transformer-chains-sourcenext "Direct link to pre-transformer-chains-sourcenext") * Entry: `source.next: "transformerId"` or `source.next: ["t1", "t2"]` * Chaining: `transformer.next: "nextId"` walks forward; array stops walking * Exit: chain ends, event reaches collector * Multiple sources can reference the same transformer (fan-in) * No `next` = source connects directly to collector ### Post-transformer chains (`destination.before`)[​](#post-transformer-chains-destinationbefore "Direct link to post-transformer-chains-destinationbefore") * Entry: `destination.before: "transformerId"` or `destination.before: ["t1", "t2"]` * Same `transformer.next` chain-walking logic as pre-chains * Exit: chain ends, event reaches destination * Multiple destinations can share the same transformer * No `before` = collector connects directly to destination ### Chain resolution algorithm (`getNextSteps`)[​](#chain-resolution-algorithm-getnextsteps "Direct link to chain-resolution-algorithm-getnextsteps") See [packages/collector/src/transformer.ts](https://github.com/elbwalker/walkerOS/blob/main/packages/collector/src/transformer.ts) for the implementation. `getNextSteps` is the public dispatch helper that replaces the previous `walkChain` entry point. * **String start:** walks `transformer.next` links until chain ends * **Array start:** uses array as-is (explicit chain, no walking) * **Array `next` inside chain:** appends array elements and stops walking * **Circular references:** detected via visited set, silently breaks loop * **Non-existent transformer ID:** chain ends (no error, event proceeds without transformation) Note: `getNextSteps` is deterministic for the supplied event context. Static analyzers without a real event can only enumerate reachability under "match may pass or fail" speculation. ### Conditional routing (`one` operator)[​](#conditional-routing-one-operator "Direct link to conditional-routing-one-operator") The `next` and `before` properties accept a `Route` (`string | Route[] | RouteConfig`). A `RouteConfig` is a **disjoint union**: each config sets at most one of `next` (gated link), `one` (first-match dispatch), or `many` (all-match dispatch), never more than one. The `one` operator enables conditional routing evaluated against ingest data and picks the first entry whose `match` succeeds: ``` "next": { "one": [ { "match": { "key": "ingest.path", "operator": "prefix", "value": "/api" }, "next": "api-handler" }, { "next": "default" } ] } ``` * `one` entries are evaluated in order, first match wins * An entry without `match` always matches, use it as the fallback * No matching entry means the event passes through unchanged * Works on all chain positions: `source.before`, `source.next`, `transformer.before`, `transformer.next`, `destination.before`, and `destination.next` * Routes are compiled to closures at init time for fast per-event evaluation * See [packages/core/src/route.ts](https://github.com/elbwalker/walkerOS/blob/main/packages/core/src/route.ts) for `compileNext()` and `resolveNext()` ### All-match dispatch (`many` operator)[​](#all-match-dispatch-many-operator "Direct link to all-match-dispatch-many-operator") Use `many` when every matching entry should produce an independent parallel flow (audit-while-process, multi-decoder fan-out). `many` terminates the main chain, each branch runs to its own exit. Available only pre-collector. Post-collector fan-out uses the destinations map. ``` "next": { "many": [ { "match": { "key": "event.consent.analytics", "operator": "eq", "value": "granted" }, "next": "ga4-pipeline" }, { "next": "audit-log" } ] } ``` ### Paths and pass-through steps (code-less transformer entries)[​](#paths-and-pass-through-steps-code-less-transformer-entries "Direct link to Paths and pass-through steps (code-less transformer entries)") A **path** is the multi-step chain through a flow's `transformers` section. A **pass-through step** (short: **pass**) is a single step inside a path that declares no `code` and no `package`; the runtime synthesizes its push from the operative fields the step does declare. Pass-through steps come in three variants: * **Chain-only:** only `before` and/or `next` set. A named hop that shares a chain across multiple call sites (avoids duplicating arrays). * **Cache-only:** only `cache` set. A dedup or short-circuit step. `cache.stop: true` at a pre-collector position halts the pipeline. * **Mapping-only:** only `mapping: Mapping.Config` set. A declarative event-to-event transform that mutates the event in-flight. ``` { "transformers": { "validateThenEnrich": { "before": ["validate", "enrich"] } }, "destinations": { "gtag": { "package": "@walkeros/web-destination-gtag", "before": "validateThenEnrich" }, "meta": { "package": "@walkeros/web-destination-meta", "before": "validateThenEnrich" } } } ``` Transformer step entries follow a **closed schema**: unknown top-level keys are validation errors, and at least one operative field (`code` / `package` / `before` / `next` / `cache` / `mapping`) must be set. See [walkeros-understanding-transformers](https://www.walkeros.io/skills/walkeros-understanding-transformers.md) for full depth on the three variants, the closed-schema rule, and the dual semantic of `mapping` at the transformer position versus the destination position. ### Transformer sharing[​](#transformer-sharing "Direct link to Transformer sharing") A single transformer can appear in both pre-chains (`source.next`) and post-chains (`destination.before`). The same transformer pool is shared; role depends on which chain references it. ### Deferred activation (`require`)[​](#deferred-activation-require "Direct link to deferred-activation-require") * `source.config.require: ["consent"]` - source deferred until "consent" event fires * `destination.config.require: ["user"]` - destination deferred until "user" event fires * Multiple requirements: all must be fulfilled (each fires independently) ### Mapping and consent gating[​](#mapping-and-consent-gating "Direct link to Mapping and consent gating") * **Source-level:** `source.config.mapping` and `source.config.consent` - applied before pre-chain; blocks event entirely * **Destination-level:** `destination.config.mapping` and `destination.config.consent` - applied after post-chain; skips only that destination, queues denied events ### Canvas rendering rules (for UI graph visualization)[​](#canvas-rendering-rules-for-ui-graph-visualization "Direct link to Canvas rendering rules (for UI graph visualization)") * **Shared transformers (pre+post):** duplicate visually with a link indicator; editing one updates the other * **Router fan-out:** keep graph planar; trace edges individually * **Orphan transformers (not in any chain):** render grey/muted; gain color when connected * **Diamond patterns (fan-in + fan-out):** expected and valid * **Overlapping `destination.before` chains:** intentional (e.g., shared validator for monitoring) ## Related Skills[​](#related-skills "Direct link to Related Skills") * [walkeros-understanding-events](https://www.walkeros.io/skills/walkeros-understanding-events.md) - Event model * [walkeros-understanding-sources](https://www.walkeros.io/skills/walkeros-understanding-sources.md) - Source interface * [walkeros-understanding-destinations](https://www.walkeros.io/skills/walkeros-understanding-destinations.md) - Destination interface **Package READMEs:** * [packages/collector/README.md](https://github.com/elbwalker/walkerOS/blob/main/packages/collector/README.md) - Collector details **Source Files:** * [packages/collector/src/flow.ts](https://github.com/elbwalker/walkerOS/blob/main/packages/collector/src/flow.ts) - startFlow implementation **Documentation:** * [Website: Flow](https://www.walkeros.io/docs/getting-started/modes/bundled.md) - Flow concept * [Website: Collector](https://www.walkeros.io/docs/collector.md) - Collector docs