AWS Firehose
Looking for AWS SNS?
Topic publishing with FIFO ordering, KMS encryption, and topic provisioning lives on the dedicated AWS SNS destination page.
Streams events from walkerOS to an Amazon Data Firehose stream, which delivers them to S3, Snowflake, Redshift, OpenSearch or an HTTP endpoint. Batching is on by default, every record ends with a newline so S3 objects are valid NDJSON, and a record AWS rejects is reported as failed with its cause. The destination needs a stream name and, off AWS, credentials.
Where this fits
AWS Firehose is a server destination in the walkerOS flow:
Streams events to Amazon Data Firehose for S3 data lakes, Snowflake, Redshift and other downstream AWS services.
Quick start
"destinations": {
"firehose": {
"package": "@walkeros/server-destination-aws",
"import": "destinationFirehose",
"config": {
"settings": { "streamName": "walkeros-events" }
}
}
}
That is the whole configuration on AWS compute in the stream's region. Add
settings.region when the stream lives elsewhere, and
credentials when the flow runs outside AWS.
Installation
npm install @walkeros/server-destination-aws- Integrated
- Bundled
import { startFlow } from '@walkeros/collector';
import { destinationFirehose } from '@walkeros/server-destination-aws';
// Credentials come from the AWS default chain (role, AWS_* variables or profile).
await startFlow({
destinations: {
firehose: {
code: destinationFirehose,
config: {
settings: {
streamName: 'walkeros-events',
region: 'eu-central-1',
},
},
},
},
});Add to your flow.json destinations:
"destinations": {
"firehose": {
"package": "@walkeros/server-destination-aws",
"import": "destinationFirehose",
"config": {
"settings": {
"streamName": "walkeros-events",
"region": "eu-central-1"
},
"credentials": {
"accessKeyId": "$secret.AWS_ACCESS_KEY_ID",
"secretAccessKey": "$secret.AWS_SECRET_ACCESS_KEY"
}
}
}
}Configuration
This destination uses the standard destination config wrapper (consent, data, env, id, ...). For the shared fields see destination configuration. Package-specific fields live under config.settings and are listed below.
Settings
| Property | Type | Description | More |
|---|---|---|---|
streamName | string | Name of the Firehose stream (like walkeros-events). Required. | |
region | string | AWS region of the stream (like eu-central-1). Default: AWS_REGION or the active profile, else eu-central-1. | |
newline | boolean | Append a newline to every record (default true). Set false when the stream adds its own new line delimiter. | |
config | any | Raw FirehoseClient options passed to the SDK (like { endpoint, maxAttempts }). Put keys in config.credentials instead. | |
client | any | A FirehoseClient of your own. Used as is and never closed by the destination. | |
firehose | firehose | Deprecated: use the flat fields streamName, region, config and client. | |
streamName | string | Name of the Firehose stream (like walkeros-events). Required. | |
client | any | A FirehoseClient of your own. Used as is and never closed by the destination. | |
region | string | AWS region of the stream (like eu-central-1). Default: AWS_REGION or the active profile, else eu-central-1. | |
config | any | Raw FirehoseClient options passed to the SDK (like { endpoint, maxAttempts }). Put keys in config.credentials instead. |
Mapping
This package does not define custom rule-level settings. For the standard rule fields (consent, condition, data, batch, name, policy) see mapping.
Examples
Page view record
A page view is sent to Firehose as one record: the full event as JSON followed by a newline.
Event
{
"name": "page view",
"data": {
"title": "Home",
"url": "https://example.com/"
},
"context": {
"dev": [
"test",
1
]
},
"globals": {
"pagegroup": "docs"
},
"custom": {
"completely": "random"
},
"user": {
"id": "us3r",
"device": "c00k13",
"session": "s3ss10n"
},
"nested": [
{
"entity": "child",
"data": {
"is": "subordinated"
}
}
],
"consent": {
"functional": true
},
"id": "ev-1700001000",
"trigger": "load",
"entity": "page",
"action": "view",
"timestamp": 1700001000,
"timing": 3.14,
"source": {
"count": 1,
"trace": "0a1b2c3d4e5f60718293a4b5c6d7e8f9",
"type": "express",
"platform": "server"
}
}Out
client.send({
"DeliveryStreamName": "walkeros-events",
"Records": [
{
"Data": {
"0": 123,
"1": 34,
"2": 110,
"3": 97,
"4": 109,
"5": 101,
"6": 34,
"7": 58,
"8": 34,
"9": 112,
"10": 97,
"11": 103,
"12": 101,
"13": 32,
"14": 118,
"15": 105,
"16": 101,
"17": 119,
"18": 34,
"19": 44,
"20": 34,
"21": 100,
"22": 97,
"23": 116,
"24": 97,
"25": 34,
"26": 58,
"27": 123,
"28": 34,
"29": 116,
"30": 105,
"31": 116,
"32": 108,
"33": 101,
"34": 34,
"35": 58,
"36": 34,
"37": 72,
"38": 111,
"39": 109,
"40": 101,
"41": 34,
"42": 44,
"43": 34,
"44": 117,
"45": 114,
"46": 108,
"47": 34,
"48": 58,
"49": 34,
"50": 104,
"51": 116,
"52": 116,
"53": 112,
"54": 115,
"55": 58,
"56": 47,
"57": 47,
"58": 101,
"59": 120,
"60": 97,
"61": 109,
"62": 112,
"63": 108,
"64": 101,
"65": 46,
"66": 99,
"67": 111,
"68": 109,
"69": 47,
"70": 34,
"71": 125,
"72": 44,
"73": 34,
"74": 99,
"75": 111,
"76": 110,
"77": 116,
"78": 101,
"79": 120,
"80": 116,
"81": 34,
"82": 58,
"83": 123,
"84": 34,
"85": 100,
"86": 101,
"87": 118,
"88": 34,
"89": 58,
"90": 91,
"91": 34,
"92": 116,
"93": 101,
"94": 115,
"95": 116,
"96": 34,
"97": 44,
"98": 49,
"99": 93,
"100": 125,
"101": 44,
"102": 34,
"103": 103,
"104": 108,
"105": 111,
"106": 98,
"107": 97,
"108": 108,
"109": 115,
"110": 34,
"111": 58,
"112": 123,
"113": 34,
"114": 112,
"115": 97,
"116": 103,
"117": 101,
"118": 103,
"119": 114,
"120": 111,
"121": 117,
"122": 112,
"123": 34,
"124": 58,
"125": 34,
"126": 100,
"127": 111,
"128": 99,
"129": 115,
"130": 34,
"131": 125,
"132": 44,
"133": 34,
"134": 99,
"135": 117,
"136": 115,
"137": 116,
"138": 111,
"139": 109,
"140": 34,
"141": 58,
"142": 123,
"143": 34,
"144": 99,
"145": 111,
"146": 109,
"147": 112,
"148": 108,
"149": 101,
"150": 116,
"151": 101,
"152": 108,
"153": 121,
"154": 34,
"155": 58,
"156": 34,
"157": 114,
"158": 97,
"159": 110,
"160": 100,
"161": 111,
"162": 109,
"163": 34,
"164": 125,
"165": 44,
"166": 34,
"167": 117,
"168": 115,
"169": 101,
"170": 114,
"171": 34,
"172": 58,
"173": 123,
"174": 34,
"175": 105,
"176": 100,
"177": 34,
"178": 58,
"179": 34,
"180": 117,
"181": 115,
"182": 51,
"183": 114,
"184": 34,
"185": 44,
"186": 34,
"187": 100,
"188": 101,
"189": 118,
"190": 105,
"191": 99,
"192": 101,
"193": 34,
"194": 58,
"195": 34,
"196": 99,
"197": 48,
"198": 48,
"199": 107,
"200": 49,
"201": 51,
"202": 34,
"203": 44,
"204": 34,
"205": 115,
"206": 101,
"207": 115,
"208": 115,
"209": 105,
"210": 111,
"211": 110,
"212": 34,
"213": 58,
"214": 34,
"215": 115,
"216": 51,
"217": 115,
"218": 115,
"219": 49,
"220": 48,
"221": 110,
"222": 34,
"223": 125,
"224": 44,
"225": 34,
"226": 110,
"227": 101,
"228": 115,
"229": 116,
"230": 101,
"231": 100,
"232": 34,
"233": 58,
"234": 91,
"235": 123,
"236": 34,
"237": 101,
"238": 110,
"239": 116,
"240": 105,
"241": 116,
"242": 121,
"243": 34,
"244": 58,
"245": 34,
"246": 99,
"247": 104,
"248": 105,
"249": 108,
"250": 100,
"251": 34,
"252": 44,
"253": 34,
"254": 100,
"255": 97,
"256": 116,
"257": 97,
"258": 34,
"259": 58,
"260": 123,
"261": 34,
"262": 105,
"263": 115,
"264": 34,
"265": 58,
"266": 34,
"267": 115,
"268": 117,
"269": 98,
"270": 111,
"271": 114,
"272": 100,
"273": 105,
"274": 110,
"275": 97,
"276": 116,
"277": 101,
"278": 100,
"279": 34,
"280": 125,
"281": 125,
"282": 93,
"283": 44,
"284": 34,
"285": 99,
"286": 111,
"287": 110,
"288": 115,
"289": 101,
"290": 110,
"291": 116,
"292": 34,
"293": 58,
"294": 123,
"295": 34,
"296": 102,
"297": 117,
"298": 110,
"299": 99,
"300": 116,
"301": 105,
"302": 111,
"303": 110,
"304": 97,
"305": 108,
"306": 34,
"307": 58,
"308": 116,
"309": 114,
"310": 117,
"311": 101,
"312": 125,
"313": 44,
"314": 34,
"315": 105,
"316": 100,
"317": 34,
"318": 58,
"319": 34,
"320": 101,
"321": 118,
"322": 45,
"323": 49,
"324": 55,
"325": 48,
"326": 48,
"327": 48,
"328": 48,
"329": 49,
"330": 48,
"331": 48,
"332": 48,
"333": 34,
"334": 44,
"335": 34,
"336": 116,
"337": 114,
"338": 105,
"339": 103,
"340": 103,
"341": 101,
"342": 114,
"343": 34,
"344": 58,
"345": 34,
"346": 108,
"347": 111,
"348": 97,
"349": 100,
"350": 34,
"351": 44,
"352": 34,
"353": 101,
"354": 110,
"355": 116,
"356": 105,
"357": 116,
"358": 121,
"359": 34,
"360": 58,
"361": 34,
"362": 112,
"363": 97,
"364": 103,
"365": 101,
"366": 34,
"367": 44,
"368": 34,
"369": 97,
"370": 99,
"371": 116,
"372": 105,
"373": 111,
"374": 110,
"375": 34,
"376": 58,
"377": 34,
"378": 118,
"379": 105,
"380": 101,
"381": 119,
"382": 34,
"383": 44,
"384": 34,
"385": 116,
"386": 105,
"387": 109,
"388": 101,
"389": 115,
"390": 116,
"391": 97,
"392": 109,
"393": 112,
"394": 34,
"395": 58,
"396": 49,
"397": 55,
"398": 48,
"399": 48,
"400": 48,
"401": 48,
"402": 49,
"403": 48,
"404": 48,
"405": 48,
"406": 44,
"407": 34,
"408": 116,
"409": 105,
"410": 109,
"411": 105,
"412": 110,
"413": 103,
"414": 34,
"415": 58,
"416": 51,
"417": 46,
"418": 49,
"419": 52,
"420": 44,
"421": 34,
"422": 115,
"423": 111,
"424": 117,
"425": 114,
"426": 99,
"427": 101,
"428": 34,
"429": 58,
"430": 123,
"431": 34,
"432": 99,
"433": 111,
"434": 117,
"435": 110,
"436": 116,
"437": 34,
"438": 58,
"439": 49,
"440": 44,
"441": 34,
"442": 116,
"443": 114,
"444": 97,
"445": 99,
"446": 101,
"447": 34,
"448": 58,
"449": 34,
"450": 48,
"451": 97,
"452": 49,
"453": 98,
"454": 50,
"455": 99,
"456": 51,
"457": 100,
"458": 52,
"459": 101,
"460": 53,
"461": 102,
"462": 54,
"463": 48,
"464": 55,
"465": 49,
"466": 56,
"467": 50,
"468": 57,
"469": 51,
"470": 97,
"471": 52,
"472": 98,
"473": 53,
"474": 99,
"475": 54,
"476": 100,
"477": 55,
"478": 101,
"479": 56,
"480": 102,
"481": 57,
"482": 34,
"483": 44,
"484": 34,
"485": 116,
"486": 121,
"487": 112,
"488": 101,
"489": 34,
"490": 58,
"491": 34,
"492": 101,
"493": 120,
"494": 112,
"495": 114,
"496": 101,
"497": 115,
"498": 115,
"499": 34,
"500": 44,
"501": 34,
"502": 112,
"503": 108,
"504": 97,
"505": 116,
"506": 102,
"507": 111,
"508": 114,
"509": 109,
"510": 34,
"511": 58,
"512": 34,
"513": 115,
"514": 101,
"515": 114,
"516": 118,
"517": 101,
"518": 114,
"519": 34,
"520": 125,
"521": 125,
"522": 10
}
}
]
})Initialization
Init builds one FirehoseClient for the stream region with the credentials from config.credentials and a 2500 ms per-attempt timeout. Nothing is sent until the first event.
Event
{
"settings": {
"streamName": "walkeros-events",
"region": "eu-central-1"
},
"credentials": {
"accessKeyId": "$secret.AWS_ACCESS_KEY_ID",
"secretAccessKey": "$secret.AWS_SECRET_ACCESS_KEY"
}
}Out
new FirehoseClient({
"region": "eu-central-1",
"credentials": {
"accessKeyId": "$secret.AWS_ACCESS_KEY_ID",
"secretAccessKey": "$secret.AWS_SECRET_ACCESS_KEY"
},
"requestHandler": {
"requestTimeout": 2500,
"throwOnRequestTimeout": true
}
})Mapped record
A mapping that produces an object sends that object as the record instead of the full event. Keep the event id in it for deduplication.
Event
{
"name": "order complete",
"data": {
"id": "ORD-400",
"total": 99.99,
"currency": "EUR"
},
"context": {
"shopping": [
"complete",
0
]
},
"globals": {
"pagegroup": "shop"
},
"custom": {
"completely": "random"
},
"user": {
"id": "us3r",
"device": "c00k13",
"session": "s3ss10n"
},
"nested": [
{
"entity": "product",
"data": {
"id": "ers",
"name": "Everyday Ruck Snack",
"color": "black",
"size": "l",
"price": 420
},
"context": {
"shopping": [
"complete",
0
]
},
"nested": []
},
{
"entity": "product",
"data": {
"id": "cc",
"name": "Cool Cap",
"size": "one size",
"price": 42
},
"context": {
"shopping": [
"complete",
0
]
},
"nested": []
},
{
"entity": "gift",
"data": {
"name": "Surprise"
},
"context": {
"shopping": [
"complete",
0
]
},
"nested": []
}
],
"consent": {
"functional": true
},
"id": "ev-1700001001",
"trigger": "load",
"entity": "order",
"action": "complete",
"timestamp": 1700001001,
"timing": 3.14,
"source": {
"count": 1,
"trace": "0a1b2c3d4e5f60718293a4b5c6d7e8f9",
"type": "express",
"platform": "server"
}
}Mapping
{
"data": {
"map": {
"id": "id",
"name": "name",
"order_id": "data.id",
"total": "data.total"
}
}
}Out
client.send({
"DeliveryStreamName": "walkeros-events",
"Records": [
{
"Data": {
"0": 123,
"1": 34,
"2": 105,
"3": 100,
"4": 34,
"5": 58,
"6": 34,
"7": 101,
"8": 118,
"9": 45,
"10": 49,
"11": 55,
"12": 48,
"13": 48,
"14": 48,
"15": 48,
"16": 49,
"17": 48,
"18": 48,
"19": 49,
"20": 34,
"21": 44,
"22": 34,
"23": 110,
"24": 97,
"25": 109,
"26": 101,
"27": 34,
"28": 58,
"29": 34,
"30": 111,
"31": 114,
"32": 100,
"33": 101,
"34": 114,
"35": 32,
"36": 99,
"37": 111,
"38": 109,
"39": 112,
"40": 108,
"41": 101,
"42": 116,
"43": 101,
"44": 34,
"45": 44,
"46": 34,
"47": 111,
"48": 114,
"49": 100,
"50": 101,
"51": 114,
"52": 95,
"53": 105,
"54": 100,
"55": 34,
"56": 58,
"57": 34,
"58": 79,
"59": 82,
"60": 68,
"61": 45,
"62": 52,
"63": 48,
"64": 48,
"65": 34,
"66": 44,
"67": 34,
"68": 116,
"69": 111,
"70": 116,
"71": 97,
"72": 108,
"73": 34,
"74": 58,
"75": 57,
"76": 57,
"77": 46,
"78": 57,
"79": 57,
"80": 125,
"81": 10
}
}
]
})Order record
An order complete event is serialized and delivered to Firehose as one newline-terminated record.
Event
{
"name": "order complete",
"data": {
"id": "ORD-400",
"total": 99.99,
"currency": "EUR"
},
"context": {
"shopping": [
"complete",
0
]
},
"globals": {
"pagegroup": "shop"
},
"custom": {
"completely": "random"
},
"user": {
"id": "us3r",
"device": "c00k13",
"session": "s3ss10n"
},
"nested": [
{
"entity": "product",
"data": {
"id": "ers",
"name": "Everyday Ruck Snack",
"color": "black",
"size": "l",
"price": 420
},
"context": {
"shopping": [
"complete",
0
]
},
"nested": []
},
{
"entity": "product",
"data": {
"id": "cc",
"name": "Cool Cap",
"size": "one size",
"price": 42
},
"context": {
"shopping": [
"complete",
0
]
},
"nested": []
},
{
"entity": "gift",
"data": {
"name": "Surprise"
},
"context": {
"shopping": [
"complete",
0
]
},
"nested": []
}
],
"consent": {
"functional": true
},
"id": "ev-1700001001",
"trigger": "load",
"entity": "order",
"action": "complete",
"timestamp": 1700001001,
"timing": 3.14,
"source": {
"count": 1,
"trace": "0a1b2c3d4e5f60718293a4b5c6d7e8f9",
"type": "express",
"platform": "server"
}
}Out
client.send({
"DeliveryStreamName": "walkeros-events",
"Records": [
{
"Data": {
"0": 123,
"1": 34,
"2": 110,
"3": 97,
"4": 109,
"5": 101,
"6": 34,
"7": 58,
"8": 34,
"9": 111,
"10": 114,
"11": 100,
"12": 101,
"13": 114,
"14": 32,
"15": 99,
"16": 111,
"17": 109,
"18": 112,
"19": 108,
"20": 101,
"21": 116,
"22": 101,
"23": 34,
"24": 44,
"25": 34,
"26": 100,
"27": 97,
"28": 116,
"29": 97,
"30": 34,
"31": 58,
"32": 123,
"33": 34,
"34": 105,
"35": 100,
"36": 34,
"37": 58,
"38": 34,
"39": 79,
"40": 82,
"41": 68,
"42": 45,
"43": 52,
"44": 48,
"45": 48,
"46": 34,
"47": 44,
"48": 34,
"49": 116,
"50": 111,
"51": 116,
"52": 97,
"53": 108,
"54": 34,
"55": 58,
"56": 57,
"57": 57,
"58": 46,
"59": 57,
"60": 57,
"61": 44,
"62": 34,
"63": 99,
"64": 117,
"65": 114,
"66": 114,
"67": 101,
"68": 110,
"69": 99,
"70": 121,
"71": 34,
"72": 58,
"73": 34,
"74": 69,
"75": 85,
"76": 82,
"77": 34,
"78": 125,
"79": 44,
"80": 34,
"81": 99,
"82": 111,
"83": 110,
"84": 116,
"85": 101,
"86": 120,
"87": 116,
"88": 34,
"89": 58,
"90": 123,
"91": 34,
"92": 115,
"93": 104,
"94": 111,
"95": 112,
"96": 112,
"97": 105,
"98": 110,
"99": 103,
"100": 34,
"101": 58,
"102": 91,
"103": 34,
"104": 99,
"105": 111,
"106": 109,
"107": 112,
"108": 108,
"109": 101,
"110": 116,
"111": 101,
"112": 34,
"113": 44,
"114": 48,
"115": 93,
"116": 125,
"117": 44,
"118": 34,
"119": 103,
"120": 108,
"121": 111,
"122": 98,
"123": 97,
"124": 108,
"125": 115,
"126": 34,
"127": 58,
"128": 123,
"129": 34,
"130": 112,
"131": 97,
"132": 103,
"133": 101,
"134": 103,
"135": 114,
"136": 111,
"137": 117,
"138": 112,
"139": 34,
"140": 58,
"141": 34,
"142": 115,
"143": 104,
"144": 111,
"145": 112,
"146": 34,
"147": 125,
"148": 44,
"149": 34,
"150": 99,
"151": 117,
"152": 115,
"153": 116,
"154": 111,
"155": 109,
"156": 34,
"157": 58,
"158": 123,
"159": 34,
"160": 99,
"161": 111,
"162": 109,
"163": 112,
"164": 108,
"165": 101,
"166": 116,
"167": 101,
"168": 108,
"169": 121,
"170": 34,
"171": 58,
"172": 34,
"173": 114,
"174": 97,
"175": 110,
"176": 100,
"177": 111,
"178": 109,
"179": 34,
"180": 125,
"181": 44,
"182": 34,
"183": 117,
"184": 115,
"185": 101,
"186": 114,
"187": 34,
"188": 58,
"189": 123,
"190": 34,
"191": 105,
"192": 100,
"193": 34,
"194": 58,
"195": 34,
"196": 117,
"197": 115,
"198": 51,
"199": 114,
"200": 34,
"201": 44,
"202": 34,
"203": 100,
"204": 101,
"205": 118,
"206": 105,
"207": 99,
"208": 101,
"209": 34,
"210": 58,
"211": 34,
"212": 99,
"213": 48,
"214": 48,
"215": 107,
"216": 49,
"217": 51,
"218": 34,
"219": 44,
"220": 34,
"221": 115,
"222": 101,
"223": 115,
"224": 115,
"225": 105,
"226": 111,
"227": 110,
"228": 34,
"229": 58,
"230": 34,
"231": 115,
"232": 51,
"233": 115,
"234": 115,
"235": 49,
"236": 48,
"237": 110,
"238": 34,
"239": 125,
"240": 44,
"241": 34,
"242": 110,
"243": 101,
"244": 115,
"245": 116,
"246": 101,
"247": 100,
"248": 34,
"249": 58,
"250": 91,
"251": 123,
"252": 34,
"253": 101,
"254": 110,
"255": 116,
"256": 105,
"257": 116,
"258": 121,
"259": 34,
"260": 58,
"261": 34,
"262": 112,
"263": 114,
"264": 111,
"265": 100,
"266": 117,
"267": 99,
"268": 116,
"269": 34,
"270": 44,
"271": 34,
"272": 100,
"273": 97,
"274": 116,
"275": 97,
"276": 34,
"277": 58,
"278": 123,
"279": 34,
"280": 105,
"281": 100,
"282": 34,
"283": 58,
"284": 34,
"285": 101,
"286": 114,
"287": 115,
"288": 34,
"289": 44,
"290": 34,
"291": 110,
"292": 97,
"293": 109,
"294": 101,
"295": 34,
"296": 58,
"297": 34,
"298": 69,
"299": 118,
"300": 101,
"301": 114,
"302": 121,
"303": 100,
"304": 97,
"305": 121,
"306": 32,
"307": 82,
"308": 117,
"309": 99,
"310": 107,
"311": 32,
"312": 83,
"313": 110,
"314": 97,
"315": 99,
"316": 107,
"317": 34,
"318": 44,
"319": 34,
"320": 99,
"321": 111,
"322": 108,
"323": 111,
"324": 114,
"325": 34,
"326": 58,
"327": 34,
"328": 98,
"329": 108,
"330": 97,
"331": 99,
"332": 107,
"333": 34,
"334": 44,
"335": 34,
"336": 115,
"337": 105,
"338": 122,
"339": 101,
"340": 34,
"341": 58,
"342": 34,
"343": 108,
"344": 34,
"345": 44,
"346": 34,
"347": 112,
"348": 114,
"349": 105,
"350": 99,
"351": 101,
"352": 34,
"353": 58,
"354": 52,
"355": 50,
"356": 48,
"357": 125,
"358": 44,
"359": 34,
"360": 99,
"361": 111,
"362": 110,
"363": 116,
"364": 101,
"365": 120,
"366": 116,
"367": 34,
"368": 58,
"369": 123,
"370": 34,
"371": 115,
"372": 104,
"373": 111,
"374": 112,
"375": 112,
"376": 105,
"377": 110,
"378": 103,
"379": 34,
"380": 58,
"381": 91,
"382": 34,
"383": 99,
"384": 111,
"385": 109,
"386": 112,
"387": 108,
"388": 101,
"389": 116,
"390": 101,
"391": 34,
"392": 44,
"393": 48,
"394": 93,
"395": 125,
"396": 44,
"397": 34,
"398": 110,
"399": 101,
"400": 115,
"401": 116,
"402": 101,
"403": 100,
"404": 34,
"405": 58,
"406": 91,
"407": 93,
"408": 125,
"409": 44,
"410": 123,
"411": 34,
"412": 101,
"413": 110,
"414": 116,
"415": 105,
"416": 116,
"417": 121,
"418": 34,
"419": 58,
"420": 34,
"421": 112,
"422": 114,
"423": 111,
"424": 100,
"425": 117,
"426": 99,
"427": 116,
"428": 34,
"429": 44,
"430": 34,
"431": 100,
"432": 97,
"433": 116,
"434": 97,
"435": 34,
"436": 58,
"437": 123,
"438": 34,
"439": 105,
"440": 100,
"441": 34,
"442": 58,
"443": 34,
"444": 99,
"445": 99,
"446": 34,
"447": 44,
"448": 34,
"449": 110,
"450": 97,
"451": 109,
"452": 101,
"453": 34,
"454": 58,
"455": 34,
"456": 67,
"457": 111,
"458": 111,
"459": 108,
"460": 32,
"461": 67,
"462": 97,
"463": 112,
"464": 34,
"465": 44,
"466": 34,
"467": 115,
"468": 105,
"469": 122,
"470": 101,
"471": 34,
"472": 58,
"473": 34,
"474": 111,
"475": 110,
"476": 101,
"477": 32,
"478": 115,
"479": 105,
"480": 122,
"481": 101,
"482": 34,
"483": 44,
"484": 34,
"485": 112,
"486": 114,
"487": 105,
"488": 99,
"489": 101,
"490": 34,
"491": 58,
"492": 52,
"493": 50,
"494": 125,
"495": 44,
"496": 34,
"497": 99,
"498": 111,
"499": 110,
"500": 116,
"501": 101,
"502": 120,
"503": 116,
"504": 34,
"505": 58,
"506": 123,
"507": 34,
"508": 115,
"509": 104,
"510": 111,
"511": 112,
"512": 112,
"513": 105,
"514": 110,
"515": 103,
"516": 34,
"517": 58,
"518": 91,
"519": 34,
"520": 99,
"521": 111,
"522": 109,
"523": 112,
"524": 108,
"525": 101,
"526": 116,
"527": 101,
"528": 34,
"529": 44,
"530": 48,
"531": 93,
"532": 125,
"533": 44,
"534": 34,
"535": 110,
"536": 101,
"537": 115,
"538": 116,
"539": 101,
"540": 100,
"541": 34,
"542": 58,
"543": 91,
"544": 93,
"545": 125,
"546": 44,
"547": 123,
"548": 34,
"549": 101,
"550": 110,
"551": 116,
"552": 105,
"553": 116,
"554": 121,
"555": 34,
"556": 58,
"557": 34,
"558": 103,
"559": 105,
"560": 102,
"561": 116,
"562": 34,
"563": 44,
"564": 34,
"565": 100,
"566": 97,
"567": 116,
"568": 97,
"569": 34,
"570": 58,
"571": 123,
"572": 34,
"573": 110,
"574": 97,
"575": 109,
"576": 101,
"577": 34,
"578": 58,
"579": 34,
"580": 83,
"581": 117,
"582": 114,
"583": 112,
"584": 114,
"585": 105,
"586": 115,
"587": 101,
"588": 34,
"589": 125,
"590": 44,
"591": 34,
"592": 99,
"593": 111,
"594": 110,
"595": 116,
"596": 101,
"597": 120,
"598": 116,
"599": 34,
"600": 58,
"601": 123,
"602": 34,
"603": 115,
"604": 104,
"605": 111,
"606": 112,
"607": 112,
"608": 105,
"609": 110,
"610": 103,
"611": 34,
"612": 58,
"613": 91,
"614": 34,
"615": 99,
"616": 111,
"617": 109,
"618": 112,
"619": 108,
"620": 101,
"621": 116,
"622": 101,
"623": 34,
"624": 44,
"625": 48,
"626": 93,
"627": 125,
"628": 44,
"629": 34,
"630": 110,
"631": 101,
"632": 115,
"633": 116,
"634": 101,
"635": 100,
"636": 34,
"637": 58,
"638": 91,
"639": 93,
"640": 125,
"641": 93,
"642": 44,
"643": 34,
"644": 99,
"645": 111,
"646": 110,
"647": 115,
"648": 101,
"649": 110,
"650": 116,
"651": 34,
"652": 58,
"653": 123,
"654": 34,
"655": 102,
"656": 117,
"657": 110,
"658": 99,
"659": 116,
"660": 105,
"661": 111,
"662": 110,
"663": 97,
"664": 108,
"665": 34,
"666": 58,
"667": 116,
"668": 114,
"669": 117,
"670": 101,
"671": 125,
"672": 44,
"673": 34,
"674": 105,
"675": 100,
"676": 34,
"677": 58,
"678": 34,
"679": 101,
"680": 118,
"681": 45,
"682": 49,
"683": 55,
"684": 48,
"685": 48,
"686": 48,
"687": 48,
"688": 49,
"689": 48,
"690": 48,
"691": 49,
"692": 34,
"693": 44,
"694": 34,
"695": 116,
"696": 114,
"697": 105,
"698": 103,
"699": 103,
"700": 101,
"701": 114,
"702": 34,
"703": 58,
"704": 34,
"705": 108,
"706": 111,
"707": 97,
"708": 100,
"709": 34,
"710": 44,
"711": 34,
"712": 101,
"713": 110,
"714": 116,
"715": 105,
"716": 116,
"717": 121,
"718": 34,
"719": 58,
"720": 34,
"721": 111,
"722": 114,
"723": 100,
"724": 101,
"725": 114,
"726": 34,
"727": 44,
"728": 34,
"729": 97,
"730": 99,
"731": 116,
"732": 105,
"733": 111,
"734": 110,
"735": 34,
"736": 58,
"737": 34,
"738": 99,
"739": 111,
"740": 109,
"741": 112,
"742": 108,
"743": 101,
"744": 116,
"745": 101,
"746": 34,
"747": 44,
"748": 34,
"749": 116,
"750": 105,
"751": 109,
"752": 101,
"753": 115,
"754": 116,
"755": 97,
"756": 109,
"757": 112,
"758": 34,
"759": 58,
"760": 49,
"761": 55,
"762": 48,
"763": 48,
"764": 48,
"765": 48,
"766": 49,
"767": 48,
"768": 48,
"769": 49,
"770": 44,
"771": 34,
"772": 116,
"773": 105,
"774": 109,
"775": 105,
"776": 110,
"777": 103,
"778": 34,
"779": 58,
"780": 51,
"781": 46,
"782": 49,
"783": 52,
"784": 44,
"785": 34,
"786": 115,
"787": 111,
"788": 117,
"789": 114,
"790": 99,
"791": 101,
"792": 34,
"793": 58,
"794": 123,
"795": 34,
"796": 99,
"797": 111,
"798": 117,
"799": 110,
"800": 116,
"801": 34,
"802": 58,
"803": 49,
"804": 44,
"805": 34,
"806": 116,
"807": 114,
"808": 97,
"809": 99,
"810": 101,
"811": 34,
"812": 58,
"813": 34,
"814": 48,
"815": 97,
"816": 49,
"817": 98,
"818": 50,
"819": 99,
"820": 51,
"821": 100,
"822": 52,
"823": 101,
"824": 53,
"825": 102,
"826": 54,
"827": 48,
"828": 55,
"829": 49,
"830": 56,
"831": 50,
"832": 57,
"833": 51,
"834": 97,
"835": 52,
"836": 98,
"837": 53,
"838": 99,
"839": 54,
"840": 100,
"841": 55,
"842": 101,
"843": 56,
"844": 102,
"845": 57,
"846": 34,
"847": 44,
"848": 34,
"849": 116,
"850": 121,
"851": 112,
"852": 101,
"853": 34,
"854": 58,
"855": 34,
"856": 101,
"857": 120,
"858": 112,
"859": 114,
"860": 101,
"861": 115,
"862": 115,
"863": 34,
"864": 44,
"865": 34,
"866": 112,
"867": 108,
"868": 97,
"869": 116,
"870": 102,
"871": 111,
"872": 114,
"873": 109,
"874": 34,
"875": 58,
"876": 34,
"877": 115,
"878": 101,
"879": 114,
"880": 118,
"881": 101,
"882": 114,
"883": 34,
"884": 125,
"885": 125,
"886": 10
}
}
]
})User signup record
A user signup event including user fields is streamed to Firehose as a JSON record.
Event
{
"name": "user signup",
"data": {
"plan": "pro",
"source": "landing-page"
},
"context": {
"dev": [
"test",
1
]
},
"globals": {
"lang": "elb"
},
"custom": {
"completely": "random"
},
"user": {
"id": "usr-789",
"email": "new@example.com"
},
"nested": [
{
"entity": "child",
"data": {
"is": "subordinated"
}
}
],
"consent": {
"functional": true
},
"id": "ev-1700001002",
"trigger": "test",
"entity": "user",
"action": "signup",
"timestamp": 1700001002,
"timing": 3.14,
"source": {
"count": 1,
"trace": "0a1b2c3d4e5f60718293a4b5c6d7e8f9",
"type": "express",
"platform": "server"
}
}Out
client.send({
"DeliveryStreamName": "walkeros-events",
"Records": [
{
"Data": {
"0": 123,
"1": 34,
"2": 110,
"3": 97,
"4": 109,
"5": 101,
"6": 34,
"7": 58,
"8": 34,
"9": 117,
"10": 115,
"11": 101,
"12": 114,
"13": 32,
"14": 115,
"15": 105,
"16": 103,
"17": 110,
"18": 117,
"19": 112,
"20": 34,
"21": 44,
"22": 34,
"23": 100,
"24": 97,
"25": 116,
"26": 97,
"27": 34,
"28": 58,
"29": 123,
"30": 34,
"31": 112,
"32": 108,
"33": 97,
"34": 110,
"35": 34,
"36": 58,
"37": 34,
"38": 112,
"39": 114,
"40": 111,
"41": 34,
"42": 44,
"43": 34,
"44": 115,
"45": 111,
"46": 117,
"47": 114,
"48": 99,
"49": 101,
"50": 34,
"51": 58,
"52": 34,
"53": 108,
"54": 97,
"55": 110,
"56": 100,
"57": 105,
"58": 110,
"59": 103,
"60": 45,
"61": 112,
"62": 97,
"63": 103,
"64": 101,
"65": 34,
"66": 125,
"67": 44,
"68": 34,
"69": 99,
"70": 111,
"71": 110,
"72": 116,
"73": 101,
"74": 120,
"75": 116,
"76": 34,
"77": 58,
"78": 123,
"79": 34,
"80": 100,
"81": 101,
"82": 118,
"83": 34,
"84": 58,
"85": 91,
"86": 34,
"87": 116,
"88": 101,
"89": 115,
"90": 116,
"91": 34,
"92": 44,
"93": 49,
"94": 93,
"95": 125,
"96": 44,
"97": 34,
"98": 103,
"99": 108,
"100": 111,
"101": 98,
"102": 97,
"103": 108,
"104": 115,
"105": 34,
"106": 58,
"107": 123,
"108": 34,
"109": 108,
"110": 97,
"111": 110,
"112": 103,
"113": 34,
"114": 58,
"115": 34,
"116": 101,
"117": 108,
"118": 98,
"119": 34,
"120": 125,
"121": 44,
"122": 34,
"123": 99,
"124": 117,
"125": 115,
"126": 116,
"127": 111,
"128": 109,
"129": 34,
"130": 58,
"131": 123,
"132": 34,
"133": 99,
"134": 111,
"135": 109,
"136": 112,
"137": 108,
"138": 101,
"139": 116,
"140": 101,
"141": 108,
"142": 121,
"143": 34,
"144": 58,
"145": 34,
"146": 114,
"147": 97,
"148": 110,
"149": 100,
"150": 111,
"151": 109,
"152": 34,
"153": 125,
"154": 44,
"155": 34,
"156": 117,
"157": 115,
"158": 101,
"159": 114,
"160": 34,
"161": 58,
"162": 123,
"163": 34,
"164": 105,
"165": 100,
"166": 34,
"167": 58,
"168": 34,
"169": 117,
"170": 115,
"171": 114,
"172": 45,
"173": 55,
"174": 56,
"175": 57,
"176": 34,
"177": 44,
"178": 34,
"179": 101,
"180": 109,
"181": 97,
"182": 105,
"183": 108,
"184": 34,
"185": 58,
"186": 34,
"187": 110,
"188": 101,
"189": 119,
"190": 64,
"191": 101,
"192": 120,
"193": 97,
"194": 109,
"195": 112,
"196": 108,
"197": 101,
"198": 46,
"199": 99,
"200": 111,
"201": 109,
"202": 34,
"203": 125,
"204": 44,
"205": 34,
"206": 110,
"207": 101,
"208": 115,
"209": 116,
"210": 101,
"211": 100,
"212": 34,
"213": 58,
"214": 91,
"215": 123,
"216": 34,
"217": 101,
"218": 110,
"219": 116,
"220": 105,
"221": 116,
"222": 121,
"223": 34,
"224": 58,
"225": 34,
"226": 99,
"227": 104,
"228": 105,
"229": 108,
"230": 100,
"231": 34,
"232": 44,
"233": 34,
"234": 100,
"235": 97,
"236": 116,
"237": 97,
"238": 34,
"239": 58,
"240": 123,
"241": 34,
"242": 105,
"243": 115,
"244": 34,
"245": 58,
"246": 34,
"247": 115,
"248": 117,
"249": 98,
"250": 111,
"251": 114,
"252": 100,
"253": 105,
"254": 110,
"255": 97,
"256": 116,
"257": 101,
"258": 100,
"259": 34,
"260": 125,
"261": 125,
"262": 93,
"263": 44,
"264": 34,
"265": 99,
"266": 111,
"267": 110,
"268": 115,
"269": 101,
"270": 110,
"271": 116,
"272": 34,
"273": 58,
"274": 123,
"275": 34,
"276": 102,
"277": 117,
"278": 110,
"279": 99,
"280": 116,
"281": 105,
"282": 111,
"283": 110,
"284": 97,
"285": 108,
"286": 34,
"287": 58,
"288": 116,
"289": 114,
"290": 117,
"291": 101,
"292": 125,
"293": 44,
"294": 34,
"295": 105,
"296": 100,
"297": 34,
"298": 58,
"299": 34,
"300": 101,
"301": 118,
"302": 45,
"303": 49,
"304": 55,
"305": 48,
"306": 48,
"307": 48,
"308": 48,
"309": 49,
"310": 48,
"311": 48,
"312": 50,
"313": 34,
"314": 44,
"315": 34,
"316": 116,
"317": 114,
"318": 105,
"319": 103,
"320": 103,
"321": 101,
"322": 114,
"323": 34,
"324": 58,
"325": 34,
"326": 116,
"327": 101,
"328": 115,
"329": 116,
"330": 34,
"331": 44,
"332": 34,
"333": 101,
"334": 110,
"335": 116,
"336": 105,
"337": 116,
"338": 121,
"339": 34,
"340": 58,
"341": 34,
"342": 117,
"343": 115,
"344": 101,
"345": 114,
"346": 34,
"347": 44,
"348": 34,
"349": 97,
"350": 99,
"351": 116,
"352": 105,
"353": 111,
"354": 110,
"355": 34,
"356": 58,
"357": 34,
"358": 115,
"359": 105,
"360": 103,
"361": 110,
"362": 117,
"363": 112,
"364": 34,
"365": 44,
"366": 34,
"367": 116,
"368": 105,
"369": 109,
"370": 101,
"371": 115,
"372": 116,
"373": 97,
"374": 109,
"375": 112,
"376": 34,
"377": 58,
"378": 49,
"379": 55,
"380": 48,
"381": 48,
"382": 48,
"383": 48,
"384": 49,
"385": 48,
"386": 48,
"387": 50,
"388": 44,
"389": 34,
"390": 116,
"391": 105,
"392": 109,
"393": 105,
"394": 110,
"395": 103,
"396": 34,
"397": 58,
"398": 51,
"399": 46,
"400": 49,
"401": 52,
"402": 44,
"403": 34,
"404": 115,
"405": 111,
"406": 117,
"407": 114,
"408": 99,
"409": 101,
"410": 34,
"411": 58,
"412": 123,
"413": 34,
"414": 99,
"415": 111,
"416": 117,
"417": 110,
"418": 116,
"419": 34,
"420": 58,
"421": 49,
"422": 44,
"423": 34,
"424": 116,
"425": 114,
"426": 97,
"427": 99,
"428": 101,
"429": 34,
"430": 58,
"431": 34,
"432": 48,
"433": 97,
"434": 49,
"435": 98,
"436": 50,
"437": 99,
"438": 51,
"439": 100,
"440": 52,
"441": 101,
"442": 53,
"443": 102,
"444": 54,
"445": 48,
"446": 55,
"447": 49,
"448": 56,
"449": 50,
"450": 57,
"451": 51,
"452": 97,
"453": 52,
"454": 98,
"455": 53,
"456": 99,
"457": 54,
"458": 100,
"459": 55,
"460": 101,
"461": 56,
"462": 102,
"463": 57,
"464": 34,
"465": 44,
"466": 34,
"467": 116,
"468": 121,
"469": 112,
"470": 101,
"471": 34,
"472": 58,
"473": 34,
"474": 101,
"475": 120,
"476": 112,
"477": 114,
"478": 101,
"479": 115,
"480": 115,
"481": 34,
"482": 44,
"483": 34,
"484": 112,
"485": 108,
"486": 97,
"487": 116,
"488": 102,
"489": 111,
"490": 114,
"491": 109,
"492": 34,
"493": 58,
"494": 34,
"495": 115,
"496": 101,
"497": 114,
"498": 118,
"499": 101,
"500": 114,
"501": 34,
"502": 125,
"503": 125,
"504": 10
}
}
]
})Region
The region is resolved once, when the destination starts, in this order:
settings.regionsettings.config.regionAWS_REGION, or theregionof the active profile (AWS_PROFILE)eu-central-1
AWS compute (Lambda, ECS, App Runner) sets AWS_REGION, so a flow running next
to its stream needs no region at all. Every delivery error names the region the
destination used.
Migrating from
settings.firehoseEarlier versions nested the options under settings.firehose. That form is
still read as an alias of the flat fields streamName, region, config and
client. When a flat field and its alias are both set to different values,
the destination refuses to start, so a half-done migration never sends events
to the wrong stream. Move the fields up one level and remove
settings.firehose.
Credentials
On AWS, configure nothing. Attach an IAM role to the runtime (ECS task role, Lambda execution role, EC2 instance profile, App Runner instance role). The AWS SDK's default credential chain picks it up, so the flow carries no keys.
Anywhere else, set config.credentials and back both values with
$secret references. Never write a literal key into a flow file.
"config": {
"settings": { "...": "..." },
"credentials": {
"accessKeyId": "$secret.AWS_ACCESS_KEY_ID",
"secretAccessKey": "$secret.AWS_SECRET_ACCESS_KEY"
}
}
config.credentials takes { accessKeyId, secretAccessKey, sessionToken? },
or the same object as a JSON string. It wins over settings.config.credentials,
which wins over the default chain. The shape is checked when the destination
starts, without contacting AWS, and error messages name the field that is wrong,
never its value. Instead of config.credentials, the standard
AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN environment
variables also work, as does AWS_PROFILE with a shared credentials file.
Kubernetes without keys: with IAM roles for service accounts (EKS) or any
OIDC web identity setup, set AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE in
the container. The default chain exchanges the token for short-lived
credentials, so the flow again carries no keys.
Credentials are loaded on the first delivery, not at start. A missing or wrong
key surfaces as the first failed event, with a message that points at
config.credentials or the environment.
IAM policy
The credentials need firehose:PutRecordBatch on the stream:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "firehose:PutRecordBatch",
"Resource": "arn:aws:firehose:<region>:<account>:deliverystream/<name>"
}
]
}
Record format
Each record is one JSON object followed by a newline (\n):
{"name":"page view","entity":"page","action":"view","id":"ev-1700001000","data":{"title":"Home","url":"https://example.com/"},...}
- Payload: the full walkerOS event, or the mapped
datawhen the event's mapping produces an object. Keep the eventidin mapped data, because it is the key for deduplication downstream (see Delivery). settings.newline:trueby default. Firehose concatenates records into S3 objects as they arrive, so without a newline an object holds}{between records and most readers fail on it.
Turn one of the two newlines off
A Firehose stream can add its own delimiter (the "New line delimiter" option of
its S3 destination settings). When that option is on, newline: false is
required: two newlines per record make Athena read a NULL row after every
record, or fail the query.
A mapping that keeps the id and a few fields:
"mapping": {
"order": {
"complete": {
"data": {
"map": {
"id": "id",
"name": "name",
"order_id": "data.id",
"total": "data.total"
}
}
}
}
}
Snowflake
Firehose delivers to Snowflake row by row and needs no delimiter, so set
newline: false for a stream with a Snowflake destination. With the stream's
option to use JSON keys as table column names, map data to an object whose
keys match the table columns:
"config": {
"settings": { "streamName": "walkeros-snowflake", "newline": false },
"mapping": {
"*": {
"*": {
"data": {
"map": {
"event_id": "id",
"event_name": "name",
"event_timestamp": "timestamp",
"event_data": "data"
}
}
}
}
}
}
Batching
Batching is on by default with config.batch: { size: 500, age: 1000 }: the
collector flushes after 500 events or one second, whichever comes first.
Override config.batch to trade latency for fewer calls. Firehose buffers
again before it writes to the target, so small batches rarely pay off.
A flushed batch goes out as PutRecordBatch calls of at most 500 records and
2,900,000 bytes each, sent concurrently. A failed call fails only its own
records; every other call still lands. Two kinds of records are refused before
anything is sent, and only that record fails:
| Code | Cause |
|---|---|
RecordTooLarge | The record is over 1,000,000 bytes, newline included. |
InvalidRecord | The payload does not serialize to JSON (a cycle, BigInt). |
Delivery
Delivery is at least once: a record can arrive twice, for example when AWS
stored it but the answer was lost on the way back. Deduplicate downstream on
the event id.
The outcome of every delivery is exact. A batch resolves only when every record was confirmed. A record AWS rejects inside an accepted call counts as failed and is reported to the collector with its index. When nothing landed, the whole batch fails with the dominant cause. Each failure is an error with:
| Field | Meaning |
|---|---|
code | The AWS error name (ServiceUnavailableException, ResourceNotFoundException, ...) or a code above. |
status | The HTTP status when AWS answered. |
retryable | Whether sending the same record again can succeed, as classified by the AWS SDK. |
The destination sends each call once. The AWS SDK's standard retry (up to three
attempts) stays on, and each attempt is bounded at a quarter of config.timeout
(2.5 seconds at the default 10 seconds), so all attempts end before the
collector stops waiting. Passing your own settings.config.requestHandler
replaces that bound.
Records rejected by AWS are not replayed yet
There is no retry beyond the SDK's own. A record that AWS throttles or rejects goes to the collector's in-memory dead letter queue, which nothing replays, so it is lost. The collector logs every failure with its cause, so the loss is visible. A standard retry for all destinations is planned in the collector.
On shutdown, the destination waits for the calls still in flight before it
closes the client it built. A client passed as settings.client is never
closed.
Troubleshooting
| Symptom | Cause and fix |
|---|---|
ResourceNotFoundException | The stream does not exist in that region. Check settings.streamName and the region the message names. |
AccessDeniedException | The credentials lack firehose:PutRecordBatch on the stream. See IAM policy. |
UnrecognizedClientException, InvalidSignatureException, ExpiredTokenException, no credentials | The keys are wrong, expired or missing. Check config.credentials or the AWS credentials in the environment. |
| NULL rows in Athena, or failed queries | Records end with two newlines. Turn one of the two newlines off: set newline: false or the stream's delimiter off. |
S3 objects contain }{ | Records have no delimiter at all. Leave newline at its default true. |
| The destination refuses to start | A configuration error found offline: missing streamName, a flat field and settings.firehose disagree, or malformed credentials. The message names the field. |
Any other AWS error passes through with its AWS text, the stream and the region.
Simulate and local testing
walkeros push --simulate destination.firehose runs offline with the package's
mock Firehose client and prints each PutRecordBatch call with its records;
nothing reaches AWS.
To send real requests to a local Firehose-compatible endpoint (such as
LocalStack), set settings.config.endpoint:
"settings": {
"streamName": "walkeros-events",
"config": { "endpoint": "http://localhost:4566" }
}
To use a named profile from your AWS config files on a workstation, set
AWS_PROFILE. Its credentials and region are picked up without any change to
the flow.
Next steps
- AWS SNS destination for topic fan-out
- Mapping configuration
- Flow configuration
💡 Need implementation support?
elbwalker offers hands-on support: setup review, measurement planning, destination mapping, and live troubleshooting. Book a 2-hour session (€399)