Logger
walkerOS has one logger for the collector, sources, and destinations. It has four log levels and scoped loggers.
Configuration
Configure logging when initializing the collector:
import { startFlow } from '@walkeros/collector';
const { collector, elb } = await startFlow({
logger: {
level: 'DEBUG', // 'ERROR' | 'WARN' | 'INFO' | 'DEBUG'
},
});Log levels
| Level | Value | Description |
|---|---|---|
ERROR | 0 | Only errors (default) |
WARN | 1 | Errors and warnings |
INFO | 2 | Errors, warnings, and informational |
DEBUG | 3 | All messages including debug information |
Choosing a level on public endpoints
A collector endpoint is exposed to the open internet, so log level decides whether ambient noise becomes an alert. The levels follow one contract:
ERROR: the intent is not met. An event was not delivered, a setting has no effect, a producer was told something false, an invalid event was rejected.WARN: walkerOS acted and the intent is still met.INFO: lifecycle and self-healing.DEBUG: the trace of each event and deliberate decisions. A skipped event writes aDEBUGline. A consent wait writes no line: the push result lists the destination underqueuedwith the reasonconsent. Neither is a failure.
| Situation | Level |
|---|---|
| Ambient internet noise: unparseable bodies, oversized payloads | debug, plus the source's rejected counter |
| Ambient internet noise: unknown paths | answered with 404, no line |
| An event the pipeline rejects as invalid | error with the reason, one line per source and minute, counted as rejected |
A settled ok: false arriving after an HTTP ack was already sent | debug; the collector's failure line reports the cause at error |
| Handler faults and rejected push promises | error |
The collector already logs thrown pipeline failures at error, so sources do
not duplicate that. Apart from the invalid-event line, nothing a scanner can
trigger should reach error.
In the complete example
The server collector of flow-complete.json, the tested example flow, takes its log level
from a flow variable:
{
"level": "$var.logLevel"
}operateUsing the logger
The logger is automatically available in destination and source contexts:
In destinations
import { StepError } from '@walkeros/core';
const myDestination = {
type: 'my-destination',
config: {},
async push(event, { logger }) {
logger.debug('Processing event', { name: event.name });
// Throw instead of logging a failure: the collector turns the throw into
// a failed push and writes the one ERROR line.
const response = await sendToService(event);
if (!response.ok)
throw new StepError(`HTTP ${response.status}`, {
code: 'transport',
status: response.status,
});
},
};A destination does not log its own failures. It throws, and the collector
writes one line per cause (see Failure lines). Use the logger
for debug traces and for notes that are not failures.
In sources
const mySource = async ({ config, env }) => {
const { logger } = env;
logger.info('Source initialized');
return {
type: 'my-source',
config,
push: async (event) => {
logger.debug('Received event', { name: event.name });
await env.push(event);
},
};
};Logger API
Methods
| Method | Description |
|---|---|
error(message, context?) | Log an error message |
warn(message, context?) | Log a warning (degraded state, config issues) |
info(message, context?) | Log an informational message |
debug(message, context?) | Log a debug message |
throw(message, context?) | Throw an exception (returns never); it writes no log line |
json(data) | Output structured JSON data |
scope(name) | Create a scoped logger with a prefixed scope |
Error handling
The logger accepts Error objects directly and automatically extracts relevant
information:
try {
await riskyOperation();
} catch (error) {
// message, name and stack are extracted automatically, and of the cause
// only its name and code
logger.error(error);
// Or with additional context
logger.error('Operation failed', { error, operation: 'sync' });
}Throwing errors
logger.throw() throws an exception in a single call and returns never. It
does not log: the collector reports the failed step once, so a throw never
produces a second line.
function validateConfig(config) {
if (!config.apiKey) {
// Throws - returns 'never' type
logger.throw('API key is required');
}
// TypeScript knows apiKey exists here due to 'never' return type
return config.apiKey;
}In a destination, a transformer or a store, prefer throwing a StepError
from @walkeros/core with a code, so the failure carries its cause (see
Create your own destination).
Failure lines
The collector writes every failure of a step, once per cause. A destination,
transformer, store or source never writes a failure line of its own. A line is
keyed by the step, the cause code and the HTTP status, and the collector writes
one line per key and minute. The first line names the counter that the repeats
are counted in, for example status.destinations.ga4.failed. Later
occurrences are counted, and the next line of that key, or the one written at
server shutdown, carries suppressed: <n>. If more than 100 different causes
are active, the extra ones are logged on every occurrence rather than dropped.
A line carries the step id, the step type, the cause code, the HTTP
status, retryable, the event name, the mapping key, a count and the error
message capped at 300 characters. It never carries event data (data, user, consent, globals,
context, custom), ingest values, vendor response bodies, setting values or
an error's cause.
The handler of the logger configuration receives every line, including these,
with the structured fields (step, code, status, retryable,
suppressed) in its context, so forwarding failures to a monitoring service
needs no parsing. Handlers add up: a flow's own handler and the runtime's both
see each line.
Failure codes
| Code | Cause | Retryable | Fix |
|---|---|---|---|
transport | A destination's request failed or threw an unknown error | yes, except a 4xx other than 429 | Check the endpoint and credentials; the status is on the line |
runtime | A destination or a store threw a TypeError, RangeError or ReferenceError with no code on its cause; a transformer's throw is transformer, a rule condition's mapping | no | Fix the code the message names; a retry fails the same way. A browser fetch rejection that a destination throws unwrapped lands here too: throw it as transport (see Reporting failures) |
timeout | A destination or transformer push or init, or a destination's rule condition, did not settle within its config.timeout (10 s by default) | yes | Raise config.timeout or check the vendor |
not-ready | The destination or transformer could not start | no | Fix the cause its message names |
breaker-open | config.breaker is set and threshold retryable failures in a row opened it; events fail at once | yes | Fix the outage. After breaker.cooldown (30 s) the next event probes the destination and a success closes the breaker; events failed while it was open stay in the dead-letter queue |
blocked | A state delivery to the destination (its on handler, for example consent) failed and did not recover within config.timeout | yes | Fix the destination's on handler (see its consent-not-applied line). A handler that failed is retried by a push after the window, one that timed out gets a new delivery once it settles, and one that never settles keeps the destination blocked; events fail until a delivery succeeds |
queue-full | A queue refused the event: the collector has not run yet, or a held destination's queue is full | yes | Start the collector sooner or raise queueMax |
mapping | A mapping rule condition threw, or a mapped value could not be built | no | Fix the condition or the value |
transformer | A transformer threw | no | Fix the transformer |
missing-input | The event has nothing this step can send (a required field is empty) | no | Map the missing value or skip the event with ignore: true |
config | A required setting is missing or invalid | no | Set it; walkeros validate finds most of these |
vendor-not-loaded | The vendor script did not load or was blocked | no | Allow the script or load it yourself |
vendor-not-ready | The vendor SDK is not ready yet | yes | Usually resolves on the next event |
vendor-timeout | The vendor script did not become ready in time | yes | Check the network and the script URL |
beacon-rejected | The browser refused a sendBeacon | only up to 64 KiB | Send smaller batches |
consent-not-applied | A destination's on('consent') handler failed | n/a | Fix the handler; the destination's events wait for one window of config.timeout, then fail with blocked |
rule-not-applied | A destination's before chain renamed an event onto another rule; that rule and its consent do not apply | n/a | Rename the event in a source chain or collector.next |
dlq-overflow | A destination's dead-letter queue was full and evicted its oldest entry | n/a | Raise dlqMax or fix the failing destination; counted in status.dropped |
on-overflow | A destination's queue of waiting on() deliveries was full and dropped the oldest | n/a | Raise the destination's own config.queueMax; counted in status.dropped |
decode | A store holds a value that cannot be decoded | no | Repair or delete the stored value; the read counts as a miss for that event |
invalid | An event was rejected as invalid input: a missing or malformed name, or an event that cannot be serialized to JSON (an ERROR) | no | Fix the producer |
A step's own StepError.code wins over these. A code outside this table with
neither a status nor a retryable option is not retryable. Unmet consent is
not in this table because it is never a failure.
Log format
By default lines print as text. On a server flow run by runneros, set
format to json to print one JSON object per line, with severity,
message, time, scope and context. Errors and warnings go to stderr, info and debug to stdout, which
platforms such as Cloud Run read as structured logs. The runner's first lines,
printed before it loads the flow (up to Health server listening), stay text.
"collector": {
"logger": { "level": "INFO", "format": "json" }
}Under runneros, a flow's own logger level (DEBUG included) and format apply to the collector's lines without --verbose; the runner's own DEBUG lines still need --verbose. A custom handler in the flow receives an originalHandler that does nothing under runneros, because the host prints every line once, masked and in the flow's format. A web flow has no output format, so walkeros validate
rejects logger.format there (WEB_LOGGER_FORMAT). Secrets are masked in both
formats, and a JSON line is capped at 32 KiB: a longer line keeps at most half the cap of its message (messageTruncated: true) and cuts its context to fit (contextTruncated: true).
Scoped loggers
Scoped loggers prefix each message with its scope, so a line shows where it came from:
// In a destination, the logger is already scoped to its type: [google-gtag]
push(event, { logger }) {
logger.info('Processing'); // Output: INFO [google-gtag] Processing
// Create nested scope
const ga4Logger = logger.scope('ga4');
ga4Logger.debug('Sending to GA4'); // Output: DEBUG [google-gtag:ga4] Sending to GA4
}Custom handler
To send logs to an external service, configure a custom handler:
import { startFlow } from '@walkeros/collector';
const { collector, elb } = await startFlow({
logger: {
level: 'INFO',
handler: (level, message, context, scope, originalHandler) => {
// Send to external logging service
loggingService.log({
level,
message,
context,
scope: scope.join(':'),
timestamp: Date.now(),
});
// Optionally call the original console handler
originalHandler(level, message, context, scope);
},
},
});The handler receives:
| Parameter | Type | Description |
|---|---|---|
level | Level | Log level enum (ERROR=0, WARN=1, INFO=2, DEBUG=3) |
message | string | The log message |
context | LogContext | Additional context object |
scope | string[] | Array of scope names |
originalHandler | DefaultHandler | The default console handler |
Testing
Use the mock logger utility for testing destinations and sources:
import { createMockLogger } from '@walkeros/core';
describe('MyDestination', () => {
it('logs events correctly', async () => {
const mockLogger = createMockLogger();
await myDestination.push(testEvent, {
config: {},
logger: mockLogger,
});
expect(mockLogger.debug).toHaveBeenCalledWith('Processing event', {
name: testEvent.name,
});
});
});The mock logger provides jest mock functions for all logger methods plus:
scopedLoggers: Array of scoped loggers created viascope()