AWS SQS
Subscribes to an Amazon SQS queue and forwards each delivered message to the walkerOS collector. It is a long-running streaming subscriber backed by SQS long-poll, with decoders for JSON, text, and raw payloads and idempotent queue provisioning with optional dead-letter wiring and SNS-topic subscription. The default region is eu-central-1.
The source ships inside @walkeros/server-source-aws alongside the Lambda handler; install the package once and import sourceSqs for SQS.
SQS is a server source in the walkerOS flow:
Pulls messages from an SQS queue at consumer pace via long-poll. Standard pattern: SNS topic fans out to one or more SQS queues; walkerOS pulls each queue independently.
Installation
npm install @walkeros/server-source-awsimport { sourceSqs } from '@walkeros/server-source-aws';
import { startFlow } from '@walkeros/collector';
await startFlow({
sources: {
sqs: {
code: sourceSqs,
config: {
settings: {
queueName: 'YOUR_QUEUE_NAME',
region: 'eu-central-1',
},
},
},
},
});Configuration
This source uses the standard source config wrapper (consent, data, env, id, ...). For the shared fields see source configuration. Package-specific fields live under config.settings and are listed below.
Settings
| Property | Type | Description | More |
|---|---|---|---|
queueName | string | SQS queue short name (like walkeros-events). Required for both setup and runtime poll. | |
region | string | AWS region (like eu-central-1). Default: eu-central-1. | |
queueUrl | string | Optional pre-resolved queue URL. When set, init skips the GetQueueUrl lookup. | |
client | any | Pre-configured AWS SQSClient instance. Bypasses construction when supplied. | |
config | any | AWS SDK SQSClientConfig (credentials, endpoint overrides, retries). | |
decoder | 'json' | 'text' | 'raw' | Decoder for the message body. json (default) parses JSON; text forwards the body as data.payload; raw forwards it base64 encoded as data.payload. text and raw events need a before transformer that sets the event name. | |
maxMessages | integer | SQS receive batch size. Cap 10. Default: 10. | |
waitTimeSeconds | integer | Long-poll duration in seconds. Cap 20. Default: 20. | |
visibilityTimeout | integer | Per-receive visibility timeout override. Default: queue-configured value. | |
shutdownTimeoutMs | integer | Graceful shutdown timeout in milliseconds. Default: 30000. After this window, destroy() force-closes. | |
onPushError | 'nack' | 'ack' | Behavior when a message is not delivered and a redelivery could still deliver it (a retryable failure), or when decoding or forwarding throws. nack (default) skips DeleteMessage so the message redelivers; ack drops it. |
Mapping
This package does not define custom rule-level settings. For the standard rule fields (consent, condition, data, batch, name, policy) see mapping.
Examples
Text decoder
When decoder is "text", the body string flows under data.payload. It has no event name, so without a before transformer that builds the event the collector rejects it and the message is acknowledged.
{
"MessageId": "m-3",
"Body": "raw text payload"
}message.ack("m-3")Malformed JSON nacks
Default decoder is JSON. A malformed body throws DecoderError; the source nacks (skips DeleteMessage) so the message redelivers.
{
"MessageId": "m-4",
"Body": "{not json"
}message.nack("m-4")Order complete from SQS
Order complete event flowing through the long-poll loop.
{
"MessageId": "m-2",
"Body": "{\"name\":\"order complete\",\"data\":{\"id\":\"ORD-500\",\"value\":199.99,\"currency\":\"EUR\"}}"
}message.ack("m-2")Page view from SQS
A standard SQS message body containing a walker elb event payload as JSON.
{
"MessageId": "m-1",
"Body": "{\"name\":\"page view\",\"data\":{\"title\":\"Documentation\",\"url\":\"https://example.com/docs\"}}"
}message.ack("m-1")Lifecycle
The SQS source follows the long-running listener pattern, same shape as the Pub/Sub pull source:
init()resolves the queue URL viaGetQueueUrland captures the queue ARN. When the collector runs, the source starts the long-poll loop as a background task.- The instance's
push()called without arguments does nothing; called with a synthetic message it runs the same handler as the loop (tests, simulation). destroy()stops the loop, drains in-flight ack/nacks, and force-closes aftershutdownTimeoutMs(default 30000).
Authentication
The AWS SDK v3 picks up credentials in this order:
- Environment variables:
AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY,AWS_REGION. Easiest for local development. - Shared credentials file (
~/.aws/credentials) and config file (~/.aws/config). Standard CLI / SDK convention. - IAM role attached to the runtime (EC2 instance profile, ECS task role, EKS service account, Lambda execution role). Recommended for production.
- Pre-configured
SQSClient: pass an instance viasettings.clientfor full control.
The runtime identity needs sqs:ReceiveMessage, sqs:DeleteMessage, and sqs:GetQueueUrl on the target queue (and sqs:GetQueueAttributes for the ARN capture at init).
Setup
In a flow file the step names the export with "import", because the package has no default export:
"sources": {
"sqs": {
"package": "@walkeros/server-source-aws",
"import": "sourceSqs",
"config": {
"settings": {
"queueName": "YOUR_QUEUE_NAME",
"region": "eu-central-1"
},
"setup": true
}
}
}walkeros setup source.sqs would provision the queue once per environment; today it cannot load the source:
walkeros setup source.sqs stops with Package @walkeros/server-source-aws has no export "sourceSqs" (or has no default export without "import") and creates nothing. Create the queue with your AWS tooling. The rest of this section describes the source's own setup function.
Setup builds its client from setup.region (default eu-central-1), not from settings.region: outside eu-central-1, set both, or the queue is created in eu-central-1 while the source polls the other region.
Idempotent: re-running on a queue whose attributes match is a no-op. If they differ, setup stops with an error and changes nothing; delete or rename the queue, then run it again. Tags and policy statements added outside setup are not touched.
config.setup:
- omitted or
false(default): no provisioning. true: provisions with safe defaults (eu-central-1, standard queue, 30-second visibility timeout).- Object form: explicit overrides:
region,fifoQueue,visibilityTimeoutSeconds(30),messageRetentionSeconds(345600, 4 days),maximumMessageSize(262144),kmsMasterKeyId,tags, and the two helpers below.
Optional one-shot helpers:
setup.deadLetterQueue:{ create: true }creates a sibling<queueName>-dlq(for a FIFO queue<base>-dlq.fifo, where<base>is the queue name without its.fifosuffix),{ arn }uses an existing one;maxReceiveCountdefaults to 5.setup.subscribeToSnsTopic:{ topicArn, rawMessageDelivery?, filterPolicy? }subscribes the queue to an SNS topic and applies the queue policy grantingsns:SendMessagefrom that topic. SetrawMessageDelivery: true: without it each message body is the SNS envelope, which carries no event name, and every message is rejected and deleted.
Setup ignores keys it does not know, and walkeros validate does not flag them: a misspelled helper creates the queue without it and logs only that the queue was created.
The provisioning identity needs sqs:CreateQueue, sqs:GetQueueAttributes, sqs:GetQueueUrl, sqs:TagQueue, and sns:Subscribe (only when subscribing to an SNS topic).
Decoders
| Decoder | Behavior |
|---|---|
json (default) | JSON.parse(body). Throws on malformed JSON. |
text | The body string as data.payload. |
raw | The body bytes as a base64 string in data.payload. |
Neither text nor raw carries an event name: without a source.before transformer that builds the event, the collector rejects it as invalid and the message is deleted. A JSON body of null is deleted with a DEBUG line only.
A decoder that throws on malformed payloads, or a push that throws, triggers onPushError: nack lets visibility expire (SQS redelivers; eventually the DLQ if wired), ack deletes the message; the decode failure is an ERROR line. A push that fails for a retryable reason (a timeout, an unavailable vendor) takes the same path. A failure a redelivery cannot fix (a missing setting, a 4xx from the vendor, a destination that is not ready) and invalid input are acknowledged and the message is deleted: the failure is in the destination's dead-letter queue and in one ERROR line from the collector, and invalid input is counted as rejected with one ERROR line per source and minute.
Backpressure
Each ReceiveMessage returns up to maxMessages (cap 10) and waits up to waitTimeSeconds (cap 20) for messages to arrive. The loop hands each message to the collector and polls again without waiting, so in-flight messages are not capped: when pushes are slow, keep visibilityTimeout above your processing time, or messages are delivered again.
For higher throughput run multiple worker instances against the same queue; SQS distributes messages across consumers automatically.
Troubleshooting
QueueDoesNotExistat init: the queue name is correct but the queue is not provisioned. Create the queue (walkeros setupcannot load this source today, see Setup); the error message still suggestswalkeros setup source.<id>.AccessDeniedonReceiveMessage: the runtime IAM identity lackssqs:ReceiveMessageon the queue. Grant the queue ARN explicitly.- Messages stuck in flight: a handler threw before deleting and visibility timeout has not yet expired. With
onPushError: 'nack'(default) this is intentional: SQS redelivers after the visibility window. TunevisibilityTimeoutto your handler's expected processing time. - Decoder throws on every message: the queue carries a different payload shape than declared. Switch the decoder (
text,raw) and add asource.beforetransformer that turns the payload into an event.
Next steps
- AWS SNS destination for the producer side. Standard pattern: SNS topic fans out to this SQS queue, subscribed with raw message delivery (see Setup).
- GCP Pub/Sub source for the GCP analog (streaming pull subscriber).
- AWS Lambda source for invocation-triggered ingestion.