> Part of the walkerOS documentation. Project overview and full index: # AWS Firehose [Server](#)[ Source code](https://github.com/elbwalker/walkerOS/tree/main/packages/server/destinations/aws)[ Package](https://www.npmjs.com/package/@walkeros/server-destination-aws) Looking for AWS SNS? Topic publishing with FIFO ordering, KMS encryption, and topic provisioning lives on the dedicated [AWS SNS destination page](/preview/pr-749/docs/destinations/server/sns.md). Streams events from walkerOS to an [Amazon Data Firehose](https://aws.amazon.com/firehose/) stream, which delivers them to S3, Snowflake, Redshift, OpenSearch or an HTTP endpoint. Batching is on by default, every record ends with a newline so S3 objects are valid NDJSON, and a record AWS rejects is reported as failed with its cause. The destination needs a stream name and, off AWS, credentials. Where this fits AWS Firehose is a **server destination** in the walkerOS flow: Streams events to Amazon Data Firehose for S3 data lakes, Snowflake, Redshift and other downstream AWS services. ## Quick start[​](#quick-start "Direct link to Quick start") ``` "destinations": { "firehose": { "package": "@walkeros/server-destination-aws", "import": "destinationFirehose", "config": { "settings": { "streamName": "walkeros-events" } } } } ``` That is the whole configuration on AWS compute in the stream's region. Add `settings.region` when the stream lives elsewhere, and [credentials](#credentials) when the flow runs outside AWS. ## Installation[​](#installation "Direct link to Installation") ```bash npm install @walkeros/server-destination-aws ``` * Integrated * Bundled ```typescript import { startFlow } from '@walkeros/collector'; import { destinationFirehose } from '@walkeros/server-destination-aws'; // Credentials come from the AWS default chain (role, AWS_* variables or profile). await startFlow({ destinations: { firehose: { code: destinationFirehose, config: { settings: { streamName: 'walkeros-events', region: 'eu-central-1', }, }, }, }, }); ``` Add to your `flow.json` destinations: ```json "destinations": { "firehose": { "package": "@walkeros/server-destination-aws", "import": "destinationFirehose", "config": { "settings": { "streamName": "walkeros-events", "region": "eu-central-1" }, "credentials": { "accessKeyId": "$secret.AWS_ACCESS_KEY_ID", "secretAccessKey": "$secret.AWS_SECRET_ACCESS_KEY" } } } } ``` [See bundled mode setup](/preview/pr-749/docs/getting-started/modes/bundled.md) | [CLI reference](/preview/pr-749/docs/apps/cli.md) ## Configuration[​](#configuration "Direct link to Configuration") This destination uses the standard destination config wrapper (consent, data, env, id, ...). For the shared fields see [destination configuration](/docs/destinations#configuration). Package-specific fields live under `config.settings` and are listed below. ## Settings[​](#settings "Direct link to Settings") | Property | Type | Description | More | | ------------ | ---------- | ---------------------------------------------------------------------------------------------------------------------- | ---- | | `streamName` | `string` | Name of the Firehose stream (like walkeros-events). Required. | | | `region` | `string` | AWS region of the stream (like eu-central-1). Default: AWS\_REGION or the active profile, else eu-central-1. | | | `newline` | `boolean` | Append a newline to every record (default true). Set false when the stream adds its own new line delimiter. | | | `config` | `any` | Raw FirehoseClient options passed to the SDK (like { endpoint, maxAttempts }). Put keys in config.credentials instead. | | | `client` | `any` | A FirehoseClient of your own. Used as is and never closed by the destination. | | | `firehose` | `firehose` | Deprecated: use the flat fields streamName, region, config and client. | | | `streamName` | `string` | Name of the Firehose stream (like walkeros-events). Required. | | | `client` | `any` | A FirehoseClient of your own. Used as is and never closed by the destination. | | | `region` | `string` | AWS region of the stream (like eu-central-1). Default: AWS\_REGION or the active profile, else eu-central-1. | | | `config` | `any` | Raw FirehoseClient options passed to the SDK (like { endpoint, maxAttempts }). Put keys in config.credentials instead. | | ## Mapping[​](#mapping "Direct link to Mapping") This package does not define custom rule-level settings. For the standard rule fields (consent, condition, data, batch, name, policy) see [mapping](/docs/mapping). ## Examples ### Page view record A page view is sent to Firehose as one record: the full event as JSON followed by a newline. Event ```json { "name": "page view", "data": { "title": "Home", "url": "https://example.com/" }, "context": { "dev": [ "test", 1 ] }, "globals": { "pagegroup": "docs" }, "custom": { "completely": "random" }, "user": { "id": "us3r", "device": "c00k13", "session": "s3ss10n" }, "nested": [ { "entity": "child", "data": { "is": "subordinated" } } ], "consent": { "functional": true }, "id": "ev-1700001000", "trigger": "load", "entity": "page", "action": "view", "timestamp": 1700001000, "timing": 3.14, "source": { "count": 1, "trace": "0a1b2c3d4e5f60718293a4b5c6d7e8f9", "type": "express", "platform": "server" } } ``` Out ```javascript client.send({ "DeliveryStreamName": "walkeros-events", "Records": [ { "Data": { "0": 123, "1": 34, "2": 110, "3": 97, "4": 109, "5": 101, "6": 34, "7": 58, "8": 34, "9": 112, "10": 97, "11": 103, "12": 101, "13": 32, "14": 118, "15": 105, "16": 101, "17": 119, "18": 34, "19": 44, "20": 34, "21": 100, "22": 97, "23": 116, "24": 97, "25": 34, "26": 58, "27": 123, "28": 34, "29": 116, "30": 105, "31": 116, "32": 108, "33": 101, "34": 34, "35": 58, "36": 34, "37": 72, "38": 111, "39": 109, "40": 101, "41": 34, "42": 44, "43": 34, "44": 117, "45": 114, "46": 108, "47": 34, "48": 58, "49": 34, "50": 104, "51": 116, "52": 116, "53": 112, "54": 115, "55": 58, "56": 47, "57": 47, "58": 101, "59": 120, "60": 97, "61": 109, "62": 112, "63": 108, "64": 101, "65": 46, "66": 99, "67": 111, "68": 109, "69": 47, "70": 34, "71": 125, "72": 44, "73": 34, "74": 99, "75": 111, "76": 110, "77": 116, "78": 101, "79": 120, "80": 116, "81": 34, "82": 58, "83": 123, "84": 34, "85": 100, "86": 101, "87": 118, "88": 34, "89": 58, "90": 91, "91": 34, "92": 116, "93": 101, "94": 115, "95": 116, "96": 34, "97": 44, "98": 49, "99": 93, "100": 125, "101": 44, "102": 34, "103": 103, "104": 108, "105": 111, "106": 98, "107": 97, "108": 108, "109": 115, "110": 34, "111": 58, "112": 123, "113": 34, "114": 112, "115": 97, "116": 103, "117": 101, "118": 103, "119": 114, "120": 111, "121": 117, "122": 112, "123": 34, "124": 58, "125": 34, "126": 100, "127": 111, "128": 99, "129": 115, "130": 34, "131": 125, "132": 44, "133": 34, "134": 99, "135": 117, "136": 115, "137": 116, "138": 111, "139": 109, "140": 34, "141": 58, "142": 123, "143": 34, "144": 99, "145": 111, "146": 109, "147": 112, "148": 108, "149": 101, "150": 116, "151": 101, "152": 108, "153": 121, "154": 34, "155": 58, "156": 34, "157": 114, "158": 97, "159": 110, "160": 100, "161": 111, "162": 109, "163": 34, "164": 125, "165": 44, "166": 34, "167": 117, "168": 115, "169": 101, "170": 114, "171": 34, "172": 58, "173": 123, "174": 34, "175": 105, "176": 100, "177": 34, "178": 58, "179": 34, "180": 117, "181": 115, "182": 51, "183": 114, "184": 34, "185": 44, "186": 34, "187": 100, "188": 101, "189": 118, "190": 105, "191": 99, "192": 101, "193": 34, "194": 58, "195": 34, "196": 99, "197": 48, "198": 48, "199": 107, "200": 49, "201": 51, "202": 34, "203": 44, "204": 34, "205": 115, "206": 101, "207": 115, "208": 115, "209": 105, "210": 111, "211": 110, "212": 34, "213": 58, "214": 34, "215": 115, "216": 51, "217": 115, "218": 115, "219": 49, "220": 48, "221": 110, "222": 34, "223": 125, "224": 44, "225": 34, "226": 110, "227": 101, "228": 115, "229": 116, "230": 101, "231": 100, "232": 34, "233": 58, "234": 91, "235": 123, "236": 34, "237": 101, "238": 110, "239": 116, "240": 105, "241": 116, "242": 121, "243": 34, "244": 58, "245": 34, "246": 99, "247": 104, "248": 105, "249": 108, "250": 100, "251": 34, "252": 44, "253": 34, "254": 100, "255": 97, "256": 116, "257": 97, "258": 34, "259": 58, "260": 123, "261": 34, "262": 105, "263": 115, "264": 34, "265": 58, "266": 34, "267": 115, "268": 117, "269": 98, "270": 111, "271": 114, "272": 100, "273": 105, "274": 110, "275": 97, "276": 116, "277": 101, "278": 100, "279": 34, "280": 125, "281": 125, "282": 93, "283": 44, "284": 34, "285": 99, "286": 111, "287": 110, "288": 115, "289": 101, "290": 110, "291": 116, "292": 34, "293": 58, "294": 123, "295": 34, "296": 102, "297": 117, "298": 110, "299": 99, "300": 116, "301": 105, "302": 111, "303": 110, "304": 97, "305": 108, "306": 34, "307": 58, "308": 116, "309": 114, "310": 117, "311": 101, "312": 125, "313": 44, "314": 34, "315": 105, "316": 100, "317": 34, "318": 58, "319": 34, "320": 101, "321": 118, "322": 45, "323": 49, "324": 55, "325": 48, "326": 48, "327": 48, "328": 48, "329": 49, "330": 48, "331": 48, "332": 48, "333": 34, "334": 44, "335": 34, "336": 116, "337": 114, "338": 105, "339": 103, "340": 103, "341": 101, "342": 114, "343": 34, "344": 58, "345": 34, "346": 108, "347": 111, "348": 97, "349": 100, "350": 34, "351": 44, "352": 34, "353": 101, "354": 110, "355": 116, "356": 105, "357": 116, "358": 121, "359": 34, "360": 58, "361": 34, "362": 112, "363": 97, "364": 103, "365": 101, "366": 34, "367": 44, "368": 34, "369": 97, "370": 99, "371": 116, "372": 105, "373": 111, "374": 110, "375": 34, "376": 58, "377": 34, "378": 118, "379": 105, "380": 101, "381": 119, "382": 34, "383": 44, "384": 34, "385": 116, "386": 105, "387": 109, "388": 101, "389": 115, "390": 116, "391": 97, "392": 109, "393": 112, "394": 34, "395": 58, "396": 49, "397": 55, "398": 48, "399": 48, "400": 48, "401": 48, "402": 49, "403": 48, "404": 48, "405": 48, "406": 44, "407": 34, "408": 116, "409": 105, "410": 109, "411": 105, "412": 110, "413": 103, "414": 34, "415": 58, "416": 51, "417": 46, "418": 49, "419": 52, "420": 44, "421": 34, "422": 115, "423": 111, "424": 117, "425": 114, "426": 99, "427": 101, "428": 34, "429": 58, "430": 123, "431": 34, "432": 99, "433": 111, "434": 117, "435": 110, "436": 116, "437": 34, "438": 58, "439": 49, "440": 44, "441": 34, "442": 116, "443": 114, "444": 97, "445": 99, "446": 101, "447": 34, "448": 58, "449": 34, "450": 48, "451": 97, "452": 49, "453": 98, "454": 50, "455": 99, "456": 51, "457": 100, "458": 52, "459": 101, "460": 53, "461": 102, "462": 54, "463": 48, "464": 55, "465": 49, "466": 56, "467": 50, "468": 57, "469": 51, "470": 97, "471": 52, "472": 98, "473": 53, "474": 99, "475": 54, "476": 100, "477": 55, "478": 101, "479": 56, "480": 102, "481": 57, "482": 34, "483": 44, "484": 34, "485": 116, "486": 121, "487": 112, "488": 101, "489": 34, "490": 58, "491": 34, "492": 101, "493": 120, "494": 112, "495": 114, "496": 101, "497": 115, "498": 115, "499": 34, "500": 44, "501": 34, "502": 112, "503": 108, "504": 97, "505": 116, "506": 102, "507": 111, "508": 114, "509": 109, "510": 34, "511": 58, "512": 34, "513": 115, "514": 101, "515": 114, "516": 118, "517": 101, "518": 114, "519": 34, "520": 125, "521": 125, "522": 10 } } ] }) ``` ### Initialization Init builds one FirehoseClient for the stream region with the credentials from config.credentials and a 2500 ms per-attempt timeout. Nothing is sent until the first event. Event ```json { "settings": { "streamName": "walkeros-events", "region": "eu-central-1" }, "credentials": { "accessKeyId": "$secret.AWS_ACCESS_KEY_ID", "secretAccessKey": "$secret.AWS_SECRET_ACCESS_KEY" } } ``` Out ```javascript new FirehoseClient({ "region": "eu-central-1", "credentials": { "accessKeyId": "$secret.AWS_ACCESS_KEY_ID", "secretAccessKey": "$secret.AWS_SECRET_ACCESS_KEY" }, "requestHandler": { "requestTimeout": 2500, "throwOnRequestTimeout": true } }) ``` ### Mapped record A mapping that produces an object sends that object as the record instead of the full event. Keep the event id in it for deduplication. Event ```json { "name": "order complete", "data": { "id": "ORD-400", "total": 99.99, "currency": "EUR" }, "context": { "shopping": [ "complete", 0 ] }, "globals": { "pagegroup": "shop" }, "custom": { "completely": "random" }, "user": { "id": "us3r", "device": "c00k13", "session": "s3ss10n" }, "nested": [ { "entity": "product", "data": { "id": "ers", "name": "Everyday Ruck Snack", "color": "black", "size": "l", "price": 420 }, "context": { "shopping": [ "complete", 0 ] }, "nested": [] }, { "entity": "product", "data": { "id": "cc", "name": "Cool Cap", "size": "one size", "price": 42 }, "context": { "shopping": [ "complete", 0 ] }, "nested": [] }, { "entity": "gift", "data": { "name": "Surprise" }, "context": { "shopping": [ "complete", 0 ] }, "nested": [] } ], "consent": { "functional": true }, "id": "ev-1700001001", "trigger": "load", "entity": "order", "action": "complete", "timestamp": 1700001001, "timing": 3.14, "source": { "count": 1, "trace": "0a1b2c3d4e5f60718293a4b5c6d7e8f9", "type": "express", "platform": "server" } } ``` Mapping ```json { "data": { "map": { "id": "id", "name": "name", "order_id": "data.id", "total": "data.total" } } } ``` Out ```javascript client.send({ "DeliveryStreamName": "walkeros-events", "Records": [ { "Data": { "0": 123, "1": 34, "2": 105, "3": 100, "4": 34, "5": 58, "6": 34, "7": 101, "8": 118, "9": 45, "10": 49, "11": 55, "12": 48, "13": 48, "14": 48, "15": 48, "16": 49, "17": 48, "18": 48, "19": 49, "20": 34, "21": 44, "22": 34, "23": 110, "24": 97, "25": 109, "26": 101, "27": 34, "28": 58, "29": 34, "30": 111, "31": 114, "32": 100, "33": 101, "34": 114, "35": 32, "36": 99, "37": 111, "38": 109, "39": 112, "40": 108, "41": 101, "42": 116, "43": 101, "44": 34, "45": 44, "46": 34, "47": 111, "48": 114, "49": 100, "50": 101, "51": 114, "52": 95, "53": 105, "54": 100, "55": 34, "56": 58, "57": 34, "58": 79, "59": 82, "60": 68, "61": 45, "62": 52, "63": 48, "64": 48, "65": 34, "66": 44, "67": 34, "68": 116, "69": 111, "70": 116, "71": 97, "72": 108, "73": 34, "74": 58, "75": 57, "76": 57, "77": 46, "78": 57, "79": 57, "80": 125, "81": 10 } } ] }) ``` ### Order record An order complete event is serialized and delivered to Firehose as one newline-terminated record. Event ```json { "name": "order complete", "data": { "id": "ORD-400", "total": 99.99, "currency": "EUR" }, "context": { "shopping": [ "complete", 0 ] }, "globals": { "pagegroup": "shop" }, "custom": { "completely": "random" }, "user": { "id": "us3r", "device": "c00k13", "session": "s3ss10n" }, "nested": [ { "entity": "product", "data": { "id": "ers", "name": "Everyday Ruck Snack", "color": "black", "size": "l", "price": 420 }, "context": { "shopping": [ "complete", 0 ] }, "nested": [] }, { "entity": "product", "data": { "id": "cc", "name": "Cool Cap", "size": "one size", "price": 42 }, "context": { "shopping": [ "complete", 0 ] }, "nested": [] }, { "entity": "gift", "data": { "name": "Surprise" }, "context": { "shopping": [ "complete", 0 ] }, "nested": [] } ], "consent": { "functional": true }, "id": "ev-1700001001", "trigger": "load", "entity": "order", "action": "complete", "timestamp": 1700001001, "timing": 3.14, "source": { "count": 1, "trace": "0a1b2c3d4e5f60718293a4b5c6d7e8f9", "type": "express", "platform": "server" } } ``` Out ```javascript client.send({ "DeliveryStreamName": "walkeros-events", "Records": [ { "Data": { "0": 123, "1": 34, "2": 110, "3": 97, "4": 109, "5": 101, "6": 34, "7": 58, "8": 34, "9": 111, "10": 114, "11": 100, "12": 101, "13": 114, "14": 32, "15": 99, "16": 111, "17": 109, "18": 112, "19": 108, "20": 101, "21": 116, "22": 101, "23": 34, "24": 44, "25": 34, "26": 100, "27": 97, "28": 116, "29": 97, "30": 34, "31": 58, "32": 123, "33": 34, "34": 105, "35": 100, "36": 34, "37": 58, "38": 34, "39": 79, "40": 82, "41": 68, "42": 45, "43": 52, "44": 48, "45": 48, "46": 34, "47": 44, "48": 34, "49": 116, "50": 111, "51": 116, "52": 97, "53": 108, "54": 34, "55": 58, "56": 57, "57": 57, "58": 46, "59": 57, "60": 57, "61": 44, "62": 34, "63": 99, "64": 117, "65": 114, "66": 114, "67": 101, "68": 110, "69": 99, "70": 121, "71": 34, "72": 58, "73": 34, "74": 69, "75": 85, "76": 82, "77": 34, "78": 125, "79": 44, "80": 34, "81": 99, "82": 111, "83": 110, "84": 116, "85": 101, "86": 120, "87": 116, "88": 34, "89": 58, "90": 123, "91": 34, "92": 115, "93": 104, "94": 111, "95": 112, "96": 112, "97": 105, "98": 110, "99": 103, "100": 34, "101": 58, "102": 91, "103": 34, "104": 99, "105": 111, "106": 109, "107": 112, "108": 108, "109": 101, "110": 116, "111": 101, "112": 34, "113": 44, "114": 48, "115": 93, "116": 125, "117": 44, "118": 34, "119": 103, "120": 108, "121": 111, "122": 98, "123": 97, "124": 108, "125": 115, "126": 34, "127": 58, "128": 123, "129": 34, "130": 112, "131": 97, "132": 103, "133": 101, "134": 103, "135": 114, "136": 111, "137": 117, "138": 112, "139": 34, "140": 58, "141": 34, "142": 115, "143": 104, "144": 111, "145": 112, "146": 34, "147": 125, "148": 44, "149": 34, "150": 99, "151": 117, "152": 115, "153": 116, "154": 111, "155": 109, "156": 34, "157": 58, "158": 123, "159": 34, "160": 99, "161": 111, "162": 109, "163": 112, "164": 108, "165": 101, "166": 116, "167": 101, "168": 108, "169": 121, "170": 34, "171": 58, "172": 34, "173": 114, "174": 97, "175": 110, "176": 100, "177": 111, "178": 109, "179": 34, "180": 125, "181": 44, "182": 34, "183": 117, "184": 115, "185": 101, "186": 114, "187": 34, "188": 58, "189": 123, "190": 34, "191": 105, "192": 100, "193": 34, "194": 58, "195": 34, "196": 117, "197": 115, "198": 51, "199": 114, "200": 34, "201": 44, "202": 34, "203": 100, "204": 101, "205": 118, "206": 105, "207": 99, "208": 101, "209": 34, "210": 58, "211": 34, "212": 99, "213": 48, "214": 48, "215": 107, "216": 49, "217": 51, "218": 34, "219": 44, "220": 34, "221": 115, "222": 101, "223": 115, "224": 115, "225": 105, "226": 111, "227": 110, "228": 34, "229": 58, "230": 34, "231": 115, "232": 51, "233": 115, "234": 115, "235": 49, "236": 48, "237": 110, "238": 34, "239": 125, "240": 44, "241": 34, "242": 110, "243": 101, "244": 115, "245": 116, "246": 101, "247": 100, "248": 34, "249": 58, "250": 91, "251": 123, "252": 34, "253": 101, "254": 110, "255": 116, "256": 105, "257": 116, "258": 121, "259": 34, "260": 58, "261": 34, "262": 112, "263": 114, "264": 111, "265": 100, "266": 117, "267": 99, "268": 116, "269": 34, "270": 44, "271": 34, "272": 100, "273": 97, "274": 116, "275": 97, "276": 34, "277": 58, "278": 123, "279": 34, "280": 105, "281": 100, "282": 34, "283": 58, "284": 34, "285": 101, "286": 114, "287": 115, "288": 34, "289": 44, "290": 34, "291": 110, "292": 97, "293": 109, "294": 101, "295": 34, "296": 58, "297": 34, "298": 69, "299": 118, "300": 101, "301": 114, "302": 121, "303": 100, "304": 97, "305": 121, "306": 32, "307": 82, "308": 117, "309": 99, "310": 107, "311": 32, "312": 83, "313": 110, "314": 97, "315": 99, "316": 107, "317": 34, "318": 44, "319": 34, "320": 99, "321": 111, "322": 108, "323": 111, "324": 114, "325": 34, "326": 58, "327": 34, "328": 98, "329": 108, "330": 97, "331": 99, "332": 107, "333": 34, "334": 44, "335": 34, "336": 115, "337": 105, "338": 122, "339": 101, "340": 34, "341": 58, "342": 34, "343": 108, "344": 34, "345": 44, "346": 34, "347": 112, "348": 114, "349": 105, "350": 99, "351": 101, "352": 34, "353": 58, "354": 52, "355": 50, "356": 48, "357": 125, "358": 44, "359": 34, "360": 99, "361": 111, "362": 110, "363": 116, "364": 101, "365": 120, "366": 116, "367": 34, "368": 58, "369": 123, "370": 34, "371": 115, "372": 104, "373": 111, "374": 112, "375": 112, "376": 105, "377": 110, "378": 103, "379": 34, "380": 58, "381": 91, "382": 34, "383": 99, "384": 111, "385": 109, "386": 112, "387": 108, "388": 101, "389": 116, "390": 101, "391": 34, "392": 44, "393": 48, "394": 93, "395": 125, "396": 44, "397": 34, "398": 110, "399": 101, "400": 115, "401": 116, "402": 101, "403": 100, "404": 34, "405": 58, "406": 91, "407": 93, "408": 125, "409": 44, "410": 123, "411": 34, "412": 101, "413": 110, "414": 116, "415": 105, "416": 116, "417": 121, "418": 34, "419": 58, "420": 34, "421": 112, "422": 114, "423": 111, "424": 100, "425": 117, "426": 99, "427": 116, "428": 34, "429": 44, "430": 34, "431": 100, "432": 97, "433": 116, "434": 97, "435": 34, "436": 58, "437": 123, "438": 34, "439": 105, "440": 100, "441": 34, "442": 58, "443": 34, "444": 99, "445": 99, "446": 34, "447": 44, "448": 34, "449": 110, "450": 97, "451": 109, "452": 101, "453": 34, "454": 58, "455": 34, "456": 67, "457": 111, "458": 111, "459": 108, "460": 32, "461": 67, "462": 97, "463": 112, "464": 34, "465": 44, "466": 34, "467": 115, "468": 105, "469": 122, "470": 101, "471": 34, "472": 58, "473": 34, "474": 111, "475": 110, "476": 101, "477": 32, "478": 115, "479": 105, "480": 122, "481": 101, "482": 34, "483": 44, "484": 34, "485": 112, "486": 114, "487": 105, "488": 99, "489": 101, "490": 34, "491": 58, "492": 52, "493": 50, "494": 125, "495": 44, "496": 34, "497": 99, "498": 111, "499": 110, "500": 116, "501": 101, "502": 120, "503": 116, "504": 34, "505": 58, "506": 123, "507": 34, "508": 115, "509": 104, "510": 111, "511": 112, "512": 112, "513": 105, "514": 110, "515": 103, "516": 34, "517": 58, "518": 91, "519": 34, "520": 99, "521": 111, "522": 109, "523": 112, "524": 108, "525": 101, "526": 116, "527": 101, "528": 34, "529": 44, "530": 48, "531": 93, "532": 125, "533": 44, "534": 34, "535": 110, "536": 101, "537": 115, "538": 116, "539": 101, "540": 100, "541": 34, "542": 58, "543": 91, "544": 93, "545": 125, "546": 44, "547": 123, "548": 34, "549": 101, "550": 110, "551": 116, "552": 105, "553": 116, "554": 121, "555": 34, "556": 58, "557": 34, "558": 103, "559": 105, "560": 102, "561": 116, "562": 34, "563": 44, "564": 34, "565": 100, "566": 97, "567": 116, "568": 97, "569": 34, "570": 58, "571": 123, "572": 34, "573": 110, "574": 97, "575": 109, "576": 101, "577": 34, "578": 58, "579": 34, "580": 83, "581": 117, "582": 114, "583": 112, "584": 114, "585": 105, "586": 115, "587": 101, "588": 34, "589": 125, "590": 44, "591": 34, "592": 99, "593": 111, "594": 110, "595": 116, "596": 101, "597": 120, "598": 116, "599": 34, "600": 58, "601": 123, "602": 34, "603": 115, "604": 104, "605": 111, "606": 112, "607": 112, "608": 105, "609": 110, "610": 103, "611": 34, "612": 58, "613": 91, "614": 34, "615": 99, "616": 111, "617": 109, "618": 112, "619": 108, "620": 101, "621": 116, "622": 101, "623": 34, "624": 44, "625": 48, "626": 93, "627": 125, "628": 44, "629": 34, "630": 110, "631": 101, "632": 115, "633": 116, "634": 101, "635": 100, "636": 34, "637": 58, "638": 91, "639": 93, "640": 125, "641": 93, "642": 44, "643": 34, "644": 99, "645": 111, "646": 110, "647": 115, "648": 101, "649": 110, "650": 116, "651": 34, "652": 58, "653": 123, "654": 34, "655": 102, "656": 117, "657": 110, "658": 99, "659": 116, "660": 105, "661": 111, "662": 110, "663": 97, "664": 108, "665": 34, "666": 58, "667": 116, "668": 114, "669": 117, "670": 101, "671": 125, "672": 44, "673": 34, "674": 105, "675": 100, "676": 34, "677": 58, "678": 34, "679": 101, "680": 118, "681": 45, "682": 49, "683": 55, "684": 48, "685": 48, "686": 48, "687": 48, "688": 49, "689": 48, "690": 48, "691": 49, "692": 34, "693": 44, "694": 34, "695": 116, "696": 114, "697": 105, "698": 103, "699": 103, "700": 101, "701": 114, "702": 34, "703": 58, "704": 34, "705": 108, "706": 111, "707": 97, "708": 100, "709": 34, "710": 44, "711": 34, "712": 101, "713": 110, "714": 116, "715": 105, "716": 116, "717": 121, "718": 34, "719": 58, "720": 34, "721": 111, "722": 114, "723": 100, "724": 101, "725": 114, "726": 34, "727": 44, "728": 34, "729": 97, "730": 99, "731": 116, "732": 105, "733": 111, "734": 110, "735": 34, "736": 58, "737": 34, "738": 99, "739": 111, "740": 109, "741": 112, "742": 108, "743": 101, "744": 116, "745": 101, "746": 34, "747": 44, "748": 34, "749": 116, "750": 105, "751": 109, "752": 101, "753": 115, "754": 116, "755": 97, "756": 109, "757": 112, "758": 34, "759": 58, "760": 49, "761": 55, "762": 48, "763": 48, "764": 48, "765": 48, "766": 49, "767": 48, "768": 48, "769": 49, "770": 44, "771": 34, "772": 116, "773": 105, "774": 109, "775": 105, "776": 110, "777": 103, "778": 34, "779": 58, "780": 51, "781": 46, "782": 49, "783": 52, "784": 44, "785": 34, "786": 115, "787": 111, "788": 117, "789": 114, "790": 99, "791": 101, "792": 34, "793": 58, "794": 123, "795": 34, "796": 99, "797": 111, "798": 117, "799": 110, "800": 116, "801": 34, "802": 58, "803": 49, "804": 44, "805": 34, "806": 116, "807": 114, "808": 97, "809": 99, "810": 101, "811": 34, "812": 58, "813": 34, "814": 48, "815": 97, "816": 49, "817": 98, "818": 50, "819": 99, "820": 51, "821": 100, "822": 52, "823": 101, "824": 53, "825": 102, "826": 54, "827": 48, "828": 55, "829": 49, "830": 56, "831": 50, "832": 57, "833": 51, "834": 97, "835": 52, "836": 98, "837": 53, "838": 99, "839": 54, "840": 100, "841": 55, "842": 101, "843": 56, "844": 102, "845": 57, "846": 34, "847": 44, "848": 34, "849": 116, "850": 121, "851": 112, "852": 101, "853": 34, "854": 58, "855": 34, "856": 101, "857": 120, "858": 112, "859": 114, "860": 101, "861": 115, "862": 115, "863": 34, "864": 44, "865": 34, "866": 112, "867": 108, "868": 97, "869": 116, "870": 102, "871": 111, "872": 114, "873": 109, "874": 34, "875": 58, "876": 34, "877": 115, "878": 101, "879": 114, "880": 118, "881": 101, "882": 114, "883": 34, "884": 125, "885": 125, "886": 10 } } ] }) ``` ### User signup record A user signup event including user fields is streamed to Firehose as a JSON record. Event ```json { "name": "user signup", "data": { "plan": "pro", "source": "landing-page" }, "context": { "dev": [ "test", 1 ] }, "globals": { "lang": "elb" }, "custom": { "completely": "random" }, "user": { "id": "usr-789", "email": "new@example.com" }, "nested": [ { "entity": "child", "data": { "is": "subordinated" } } ], "consent": { "functional": true }, "id": "ev-1700001002", "trigger": "test", "entity": "user", "action": "signup", "timestamp": 1700001002, "timing": 3.14, "source": { "count": 1, "trace": "0a1b2c3d4e5f60718293a4b5c6d7e8f9", "type": "express", "platform": "server" } } ``` Out ```javascript client.send({ "DeliveryStreamName": "walkeros-events", "Records": [ { "Data": { "0": 123, "1": 34, "2": 110, "3": 97, "4": 109, "5": 101, "6": 34, "7": 58, "8": 34, "9": 117, "10": 115, "11": 101, "12": 114, "13": 32, "14": 115, "15": 105, "16": 103, "17": 110, "18": 117, "19": 112, "20": 34, "21": 44, "22": 34, "23": 100, "24": 97, "25": 116, "26": 97, "27": 34, "28": 58, "29": 123, "30": 34, "31": 112, "32": 108, "33": 97, "34": 110, "35": 34, "36": 58, "37": 34, "38": 112, "39": 114, "40": 111, "41": 34, "42": 44, "43": 34, "44": 115, "45": 111, "46": 117, "47": 114, "48": 99, "49": 101, "50": 34, "51": 58, "52": 34, "53": 108, "54": 97, "55": 110, "56": 100, "57": 105, "58": 110, "59": 103, "60": 45, "61": 112, "62": 97, "63": 103, "64": 101, "65": 34, "66": 125, "67": 44, "68": 34, "69": 99, "70": 111, "71": 110, "72": 116, "73": 101, "74": 120, "75": 116, "76": 34, "77": 58, "78": 123, "79": 34, "80": 100, "81": 101, "82": 118, "83": 34, "84": 58, "85": 91, "86": 34, "87": 116, "88": 101, "89": 115, "90": 116, "91": 34, "92": 44, "93": 49, "94": 93, "95": 125, "96": 44, "97": 34, "98": 103, "99": 108, "100": 111, "101": 98, "102": 97, "103": 108, "104": 115, "105": 34, "106": 58, "107": 123, "108": 34, "109": 108, "110": 97, "111": 110, "112": 103, "113": 34, "114": 58, "115": 34, "116": 101, "117": 108, "118": 98, "119": 34, "120": 125, "121": 44, "122": 34, "123": 99, "124": 117, "125": 115, "126": 116, "127": 111, "128": 109, "129": 34, "130": 58, "131": 123, "132": 34, "133": 99, "134": 111, "135": 109, "136": 112, "137": 108, "138": 101, "139": 116, "140": 101, "141": 108, "142": 121, "143": 34, "144": 58, "145": 34, "146": 114, "147": 97, "148": 110, "149": 100, "150": 111, "151": 109, "152": 34, "153": 125, "154": 44, "155": 34, "156": 117, "157": 115, "158": 101, "159": 114, "160": 34, "161": 58, "162": 123, "163": 34, "164": 105, "165": 100, "166": 34, "167": 58, "168": 34, "169": 117, "170": 115, "171": 114, "172": 45, "173": 55, "174": 56, "175": 57, "176": 34, "177": 44, "178": 34, "179": 101, "180": 109, "181": 97, "182": 105, "183": 108, "184": 34, "185": 58, "186": 34, "187": 110, "188": 101, "189": 119, "190": 64, "191": 101, "192": 120, "193": 97, "194": 109, "195": 112, "196": 108, "197": 101, "198": 46, "199": 99, "200": 111, "201": 109, "202": 34, "203": 125, "204": 44, "205": 34, "206": 110, "207": 101, "208": 115, "209": 116, "210": 101, "211": 100, "212": 34, "213": 58, "214": 91, "215": 123, "216": 34, "217": 101, "218": 110, "219": 116, "220": 105, "221": 116, "222": 121, "223": 34, "224": 58, "225": 34, "226": 99, "227": 104, "228": 105, "229": 108, "230": 100, "231": 34, "232": 44, "233": 34, "234": 100, "235": 97, "236": 116, "237": 97, "238": 34, "239": 58, "240": 123, "241": 34, "242": 105, "243": 115, "244": 34, "245": 58, "246": 34, "247": 115, "248": 117, "249": 98, "250": 111, "251": 114, "252": 100, "253": 105, "254": 110, "255": 97, "256": 116, "257": 101, "258": 100, "259": 34, "260": 125, "261": 125, "262": 93, "263": 44, "264": 34, "265": 99, "266": 111, "267": 110, "268": 115, "269": 101, "270": 110, "271": 116, "272": 34, "273": 58, "274": 123, "275": 34, "276": 102, "277": 117, "278": 110, "279": 99, "280": 116, "281": 105, "282": 111, "283": 110, "284": 97, "285": 108, "286": 34, "287": 58, "288": 116, "289": 114, "290": 117, "291": 101, "292": 125, "293": 44, "294": 34, "295": 105, "296": 100, "297": 34, "298": 58, "299": 34, "300": 101, "301": 118, "302": 45, "303": 49, "304": 55, "305": 48, "306": 48, "307": 48, "308": 48, "309": 49, "310": 48, "311": 48, "312": 50, "313": 34, "314": 44, "315": 34, "316": 116, "317": 114, "318": 105, "319": 103, "320": 103, "321": 101, "322": 114, "323": 34, "324": 58, "325": 34, "326": 116, "327": 101, "328": 115, "329": 116, "330": 34, "331": 44, "332": 34, "333": 101, "334": 110, "335": 116, "336": 105, "337": 116, "338": 121, "339": 34, "340": 58, "341": 34, "342": 117, "343": 115, "344": 101, "345": 114, "346": 34, "347": 44, "348": 34, "349": 97, "350": 99, "351": 116, "352": 105, "353": 111, "354": 110, "355": 34, "356": 58, "357": 34, "358": 115, "359": 105, "360": 103, "361": 110, "362": 117, "363": 112, "364": 34, "365": 44, "366": 34, "367": 116, "368": 105, "369": 109, "370": 101, "371": 115, "372": 116, "373": 97, "374": 109, "375": 112, "376": 34, "377": 58, "378": 49, "379": 55, "380": 48, "381": 48, "382": 48, "383": 48, "384": 49, "385": 48, "386": 48, "387": 50, "388": 44, "389": 34, "390": 116, "391": 105, "392": 109, "393": 105, "394": 110, "395": 103, "396": 34, "397": 58, "398": 51, "399": 46, "400": 49, "401": 52, "402": 44, "403": 34, "404": 115, "405": 111, "406": 117, "407": 114, "408": 99, "409": 101, "410": 34, "411": 58, "412": 123, "413": 34, "414": 99, "415": 111, "416": 117, "417": 110, "418": 116, "419": 34, "420": 58, "421": 49, "422": 44, "423": 34, "424": 116, "425": 114, "426": 97, "427": 99, "428": 101, "429": 34, "430": 58, "431": 34, "432": 48, "433": 97, "434": 49, "435": 98, "436": 50, "437": 99, "438": 51, "439": 100, "440": 52, "441": 101, "442": 53, "443": 102, "444": 54, "445": 48, "446": 55, "447": 49, "448": 56, "449": 50, "450": 57, "451": 51, "452": 97, "453": 52, "454": 98, "455": 53, "456": 99, "457": 54, "458": 100, "459": 55, "460": 101, "461": 56, "462": 102, "463": 57, "464": 34, "465": 44, "466": 34, "467": 116, "468": 121, "469": 112, "470": 101, "471": 34, "472": 58, "473": 34, "474": 101, "475": 120, "476": 112, "477": 114, "478": 101, "479": 115, "480": 115, "481": 34, "482": 44, "483": 34, "484": 112, "485": 108, "486": 97, "487": 116, "488": 102, "489": 111, "490": 114, "491": 109, "492": 34, "493": 58, "494": 34, "495": 115, "496": 101, "497": 114, "498": 118, "499": 101, "500": 114, "501": 34, "502": 125, "503": 125, "504": 10 } } ] }) ``` ## Region[​](#region "Direct link to Region") The region is resolved once, when the destination starts, in this order: 1. `settings.region` 2. `settings.config.region` 3. `AWS_REGION`, or the `region` of the active profile (`AWS_PROFILE`) 4. `eu-central-1` AWS compute (Lambda, ECS, App Runner) sets `AWS_REGION`, so a flow running next to its stream needs no region at all. Every delivery error names the region the destination used. Migrating from `settings.firehose` Earlier versions nested the options under `settings.firehose`. That form is still read as an alias of the flat fields `streamName`, `region`, `config` and `client`. When a flat field and its alias are both set to different values, the destination refuses to start, so a half-done migration never sends events to the wrong stream. Move the fields up one level and remove `settings.firehose`. ## Credentials[​](#credentials "Direct link to Credentials") **On AWS, configure nothing.** Attach an IAM role to the runtime (ECS task role, Lambda execution role, EC2 instance profile, App Runner instance role). The AWS SDK's default credential chain picks it up, so the flow carries no keys. **Anywhere else, set `config.credentials`** and back both values with `$secret` references. Never write a literal key into a flow file. ``` "config": { "settings": { "...": "..." }, "credentials": { "accessKeyId": "$secret.AWS_ACCESS_KEY_ID", "secretAccessKey": "$secret.AWS_SECRET_ACCESS_KEY" } } ``` `config.credentials` takes `{ accessKeyId, secretAccessKey, sessionToken? }`, or the same object as a JSON string. It wins over `settings.config.credentials`, which wins over the default chain. The shape is checked when the destination starts, without contacting AWS, and error messages name the field that is wrong, never its value. Instead of `config.credentials`, the standard `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_SESSION_TOKEN` environment variables also work, as does `AWS_PROFILE` with a shared credentials file. **Kubernetes without keys:** with IAM roles for service accounts (EKS) or any OIDC web identity setup, set `AWS_ROLE_ARN` and `AWS_WEB_IDENTITY_TOKEN_FILE` in the container. The default chain exchanges the token for short-lived credentials, so the flow again carries no keys. Credentials are loaded on the first delivery, not at start. A missing or wrong key surfaces as the first failed event, with a message that points at `config.credentials` or the environment. ### IAM policy[​](#iam-policy "Direct link to IAM policy") The credentials need `firehose:PutRecordBatch` on the stream: ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "firehose:PutRecordBatch", "Resource": "arn:aws:firehose:::deliverystream/" } ] } ``` ## Record format[​](#record-format "Direct link to Record format") Each record is one JSON object followed by a newline (`\n`): ``` {"name":"page view","entity":"page","action":"view","id":"ev-1700001000","data":{"title":"Home","url":"https://example.com/"},...} ``` * **Payload:** the full walkerOS event, or the mapped `data` when the event's mapping produces an object. Keep the event `id` in mapped data, because it is the key for deduplication downstream (see [Delivery](#delivery)). * **`settings.newline`:** `true` by default. Firehose concatenates records into S3 objects as they arrive, so without a newline an object holds `}{` between records and most readers fail on it. Turn one of the two newlines off A Firehose stream can add its own delimiter (the "New line delimiter" option of its S3 destination settings). When that option is on, `newline: false` is required: two newlines per record make Athena read a NULL row after every record, or fail the query. A mapping that keeps the id and a few fields: ``` "mapping": { "order": { "complete": { "data": { "map": { "id": "id", "name": "name", "order_id": "data.id", "total": "data.total" } } } } } ``` ### Snowflake[​](#snowflake "Direct link to Snowflake") Firehose delivers to Snowflake row by row and needs no delimiter, so set `newline: false` for a stream with a Snowflake destination. With the stream's option to use JSON keys as table column names, map `data` to an object whose keys match the table columns: ``` "config": { "settings": { "streamName": "walkeros-snowflake", "newline": false }, "mapping": { "*": { "*": { "data": { "map": { "event_id": "id", "event_name": "name", "event_timestamp": "timestamp", "event_data": "data" } } } } } } ``` ## Batching[​](#batching "Direct link to Batching") Batching is on by default with `config.batch: { size: 500, age: 1000 }`: the collector flushes after 500 events or one second, whichever comes first. Override `config.batch` to trade latency for fewer calls. Firehose buffers again before it writes to the target, so small batches rarely pay off. A flushed batch goes out as `PutRecordBatch` calls of at most 500 records and 2,900,000 bytes each, sent concurrently. A failed call fails only its own records; every other call still lands. Two kinds of records are refused before anything is sent, and only that record fails: | Code | Cause | | ---------------- | --------------------------------------------------------- | | `RecordTooLarge` | The record is over 1,000,000 bytes, newline included. | | `InvalidRecord` | The payload does not serialize to JSON (a cycle, BigInt). | ## Delivery[​](#delivery "Direct link to Delivery") Delivery is **at least once**: a record can arrive twice, for example when AWS stored it but the answer was lost on the way back. Deduplicate downstream on the event `id`. The outcome of every delivery is exact. A batch resolves only when every record was confirmed. A record AWS rejects inside an accepted call counts as failed and is reported to the collector with its index. When nothing landed, the whole batch fails with the dominant cause. Each failure is an error with: | Field | Meaning | | ----------- | ----------------------------------------------------------------------------------------------------- | | `code` | The AWS error name (`ServiceUnavailableException`, `ResourceNotFoundException`, ...) or a code above. | | `status` | The HTTP status when AWS answered. | | `retryable` | Whether sending the same record again can succeed, as classified by the AWS SDK. | The destination sends each call once. The AWS SDK's standard retry (up to three attempts) stays on, and each attempt is bounded at a quarter of `config.timeout` (2.5 seconds at the default 10 seconds), so all attempts end before the collector stops waiting. Passing your own `settings.config.requestHandler` replaces that bound. Records rejected by AWS are not replayed yet There is no retry beyond the SDK's own. A record that AWS throttles or rejects goes to the collector's in-memory dead letter queue, which nothing replays, so it is lost. The collector logs every failure with its cause, so the loss is visible. A standard retry for all destinations is planned in the collector. On shutdown, the destination waits for the calls still in flight before it closes the client it built. A client passed as `settings.client` is never closed. ## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") | Symptom | Cause and fix | | --------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ResourceNotFoundException` | The stream does not exist in that region. Check `settings.streamName` and the region the message names. | | `AccessDeniedException` | The credentials lack `firehose:PutRecordBatch` on the stream. See [IAM policy](#iam-policy). | | `UnrecognizedClientException`, `InvalidSignatureException`, `ExpiredTokenException`, no credentials | The keys are wrong, expired or missing. Check `config.credentials` or the AWS credentials in the environment. | | NULL rows in Athena, or failed queries | Records end with two newlines. Turn one of the two newlines off: set `newline: false` or the stream's delimiter off. | | S3 objects contain `}{` | Records have no delimiter at all. Leave `newline` at its default `true`. | | The destination refuses to start | A configuration error found offline: missing `streamName`, a flat field and `settings.firehose` disagree, or malformed credentials. The message names the field. | Any other AWS error passes through with its AWS text, the stream and the region. ## Simulate and local testing[​](#simulate-and-local-testing "Direct link to Simulate and local testing") `walkeros push --simulate destination.firehose` runs offline with the package's mock Firehose client and prints each `PutRecordBatch` call with its records; nothing reaches AWS. To send real requests to a local Firehose-compatible endpoint (such as LocalStack), set `settings.config.endpoint`: ``` "settings": { "streamName": "walkeros-events", "config": { "endpoint": "http://localhost:4566" } } ``` To use a named profile from your AWS config files on a workstation, set `AWS_PROFILE`. Its credentials and region are picked up without any change to the flow. ## Next steps[​](#next-steps "Direct link to Next steps") * [AWS SNS destination](/preview/pr-749/docs/destinations/server/sns.md) for topic fan-out * [Mapping configuration](/preview/pr-749/docs/mapping.md) * [Flow configuration](/preview/pr-749/docs/getting-started/flow.md)