Skip to main content
Ask your AI

AWS SQS

Server Source code Package

Subscribes to an Amazon SQS queue and forwards each delivered message to the walkerOS collector. It is a long-running streaming subscriber backed by SQS long-poll, with decoders for JSON, text, and raw payloads and idempotent queue provisioning with optional dead-letter wiring and SNS-topic subscription. The default region is eu-central-1.

The source ships inside @walkeros/server-source-aws alongside the Lambda handler; install the package once and import sourceSqs for SQS.

Where this fits

SQS is a server source in the walkerOS flow:

Pulls messages from an SQS queue at consumer pace via long-poll. Standard pattern: SNS topic fans out to one or more SQS queues; walkerOS pulls each queue independently.

Installation​

npm install @walkeros/server-source-aws
import { sourceSqs } from '@walkeros/server-source-aws';
import { startFlow } from '@walkeros/collector';

await startFlow({
  sources: {
    sqs: {
      code: sourceSqs,
      config: {
        settings: {
          queueName: 'YOUR_QUEUE_NAME',
          region: 'eu-central-1',
        },
      },
    },
  },
});

Configuration​

This source uses the standard source config wrapper (consent, data, env, id, ...). For the shared fields see source configuration. Package-specific fields live under config.settings and are listed below.

Settings​

PropertyTypeDescriptionMore
queueName*stringSQS queue short name (like walkeros-events). Required for both setup and runtime poll.
regionstringAWS region (like eu-central-1). Default: eu-central-1.
queueUrlstringOptional pre-resolved queue URL. When set, init skips the GetQueueUrl lookup.
clientanyPre-configured AWS SQSClient instance. Bypasses construction when supplied.
configanyAWS SDK SQSClientConfig (credentials, endpoint overrides, retries).
decoder'json' | 'text' | 'raw'Decoder for the message body. json (default) parses JSON; text forwards the body as data.payload; raw forwards it base64 encoded as data.payload. text and raw events need a before transformer that sets the event name.
maxMessagesintegerSQS receive batch size. Cap 10. Default: 10.
waitTimeSecondsintegerLong-poll duration in seconds. Cap 20. Default: 20.
visibilityTimeoutintegerPer-receive visibility timeout override. Default: queue-configured value.
shutdownTimeoutMsintegerGraceful shutdown timeout in milliseconds. Default: 30000. After this window, destroy() force-closes.
onPushError'nack' | 'ack'Behavior when a message is not delivered and a redelivery could still deliver it (a retryable failure), or when decoding or forwarding throws. nack (default) skips DeleteMessage so the message redelivers; ack drops it.
* Required fields

Mapping​

This package does not define custom rule-level settings. For the standard rule fields (consent, condition, data, batch, name, policy) see mapping.

Examples

Text decoder

When decoder is "text", the body string flows under data.payload. It has no event name, so without a before transformer that builds the event the collector rejects it and the message is acknowledged.

Event
{
  "MessageId": "m-3",
  "Body": "raw text payload"
}
Out
message.ack("m-3")

Malformed JSON nacks

Default decoder is JSON. A malformed body throws DecoderError; the source nacks (skips DeleteMessage) so the message redelivers.

Event
{
  "MessageId": "m-4",
  "Body": "{not json"
}
Out
message.nack("m-4")

Order complete from SQS

Order complete event flowing through the long-poll loop.

Event
{
  "MessageId": "m-2",
  "Body": "{\"name\":\"order complete\",\"data\":{\"id\":\"ORD-500\",\"value\":199.99,\"currency\":\"EUR\"}}"
}
Out
message.ack("m-2")

Page view from SQS

A standard SQS message body containing a walker elb event payload as JSON.

Event
{
  "MessageId": "m-1",
  "Body": "{\"name\":\"page view\",\"data\":{\"title\":\"Documentation\",\"url\":\"https://example.com/docs\"}}"
}
Out
message.ack("m-1")

Lifecycle​

The SQS source follows the long-running listener pattern, same shape as the Pub/Sub pull source:

  1. init() resolves the queue URL via GetQueueUrl and captures the queue ARN. When the collector runs, the source starts the long-poll loop as a background task.
  2. The instance's push() called without arguments does nothing; called with a synthetic message it runs the same handler as the loop (tests, simulation).
  3. destroy() stops the loop, drains in-flight ack/nacks, and force-closes after shutdownTimeoutMs (default 30000).

Authentication​

The AWS SDK v3 picks up credentials in this order:

  1. Environment variables: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION. Easiest for local development.
  2. Shared credentials file (~/.aws/credentials) and config file (~/.aws/config). Standard CLI / SDK convention.
  3. IAM role attached to the runtime (EC2 instance profile, ECS task role, EKS service account, Lambda execution role). Recommended for production.
  4. Pre-configured SQSClient: pass an instance via settings.client for full control.

The runtime identity needs sqs:ReceiveMessage, sqs:DeleteMessage, and sqs:GetQueueUrl on the target queue (and sqs:GetQueueAttributes for the ARN capture at init).

Setup​

In a flow file the step names the export with "import", because the package has no default export:

"sources": {
  "sqs": {
    "package": "@walkeros/server-source-aws",
    "import": "sourceSqs",
    "config": {
      "settings": {
        "queueName": "YOUR_QUEUE_NAME",
        "region": "eu-central-1"
      },
      "setup": true
    }
  }
}

walkeros setup source.sqs would provision the queue once per environment; today it cannot load the source:

walkeros setup cannot load this source today

walkeros setup source.sqs stops with Package @walkeros/server-source-aws has no export "sourceSqs" (or has no default export without "import") and creates nothing. Create the queue with your AWS tooling. The rest of this section describes the source's own setup function.

Setup builds its client from setup.region (default eu-central-1), not from settings.region: outside eu-central-1, set both, or the queue is created in eu-central-1 while the source polls the other region.

Idempotent: re-running on a queue whose attributes match is a no-op. If they differ, setup stops with an error and changes nothing; delete or rename the queue, then run it again. Tags and policy statements added outside setup are not touched.

config.setup:

  • omitted or false (default): no provisioning.
  • true: provisions with safe defaults (eu-central-1, standard queue, 30-second visibility timeout).
  • Object form: explicit overrides: region, fifoQueue, visibilityTimeoutSeconds (30), messageRetentionSeconds (345600, 4 days), maximumMessageSize (262144), kmsMasterKeyId, tags, and the two helpers below.

Optional one-shot helpers:

  • setup.deadLetterQueue: { create: true } creates a sibling <queueName>-dlq (for a FIFO queue <base>-dlq.fifo, where <base> is the queue name without its .fifo suffix), { arn } uses an existing one; maxReceiveCount defaults to 5.
  • setup.subscribeToSnsTopic: { topicArn, rawMessageDelivery?, filterPolicy? } subscribes the queue to an SNS topic and applies the queue policy granting sns:SendMessage from that topic. Set rawMessageDelivery: true: without it each message body is the SNS envelope, which carries no event name, and every message is rejected and deleted.

Setup ignores keys it does not know, and walkeros validate does not flag them: a misspelled helper creates the queue without it and logs only that the queue was created.

The provisioning identity needs sqs:CreateQueue, sqs:GetQueueAttributes, sqs:GetQueueUrl, sqs:TagQueue, and sns:Subscribe (only when subscribing to an SNS topic).

Decoders​

DecoderBehavior
json (default)JSON.parse(body). Throws on malformed JSON.
textThe body string as data.payload.
rawThe body bytes as a base64 string in data.payload.

Neither text nor raw carries an event name: without a source.before transformer that builds the event, the collector rejects it as invalid and the message is deleted. A JSON body of null is deleted with a DEBUG line only.

A decoder that throws on malformed payloads, or a push that throws, triggers onPushError: nack lets visibility expire (SQS redelivers; eventually the DLQ if wired), ack deletes the message; the decode failure is an ERROR line. A push that fails for a retryable reason (a timeout, an unavailable vendor) takes the same path. A failure a redelivery cannot fix (a missing setting, a 4xx from the vendor, a destination that is not ready) and invalid input are acknowledged and the message is deleted: the failure is in the destination's dead-letter queue and in one ERROR line from the collector, and invalid input is counted as rejected with one ERROR line per source and minute.

Backpressure​

Each ReceiveMessage returns up to maxMessages (cap 10) and waits up to waitTimeSeconds (cap 20) for messages to arrive. The loop hands each message to the collector and polls again without waiting, so in-flight messages are not capped: when pushes are slow, keep visibilityTimeout above your processing time, or messages are delivered again.

For higher throughput run multiple worker instances against the same queue; SQS distributes messages across consumers automatically.

Troubleshooting​

  • QueueDoesNotExist at init: the queue name is correct but the queue is not provisioned. Create the queue (walkeros setup cannot load this source today, see Setup); the error message still suggests walkeros setup source.<id>.
  • AccessDenied on ReceiveMessage: the runtime IAM identity lacks sqs:ReceiveMessage on the queue. Grant the queue ARN explicitly.
  • Messages stuck in flight: a handler threw before deleting and visibility timeout has not yet expired. With onPushError: 'nack' (default) this is intentional: SQS redelivers after the visibility window. Tune visibilityTimeout to your handler's expected processing time.
  • Decoder throws on every message: the queue carries a different payload shape than declared. Switch the decoder (text, raw) and add a source.before transformer that turns the payload into an event.

Next steps​

  • AWS SNS destination for the producer side. Standard pattern: SNS topic fans out to this SQS queue, subscribed with raw message delivery (see Setup).
  • GCP Pub/Sub source for the GCP analog (streaming pull subscriber).
  • AWS Lambda source for invocation-triggered ingestion.
💡 Need implementation support?
elbwalker offers hands-on support: setup review, measurement planning, destination mapping, and live troubleshooting. Book a 2-hour session (€399)