> Part of the walkerOS documentation. Project overview and full index: # Bot detection [Server](#)[ ](https://github.com/elbwalker/walkerOS/tree/main/packages/server/transformers/bot) [Source code](https://github.com/elbwalker/walkerOS/tree/main/packages/server/transformers/bot)[ ](https://www.npmjs.com/package/@walkeros/server-transformer-bot) [Package](https://www.npmjs.com/package/@walkeros/server-transformer-bot) Annotates events with `user.botScore` (0-99, higher = more automated), `user.botCategory` (what kind of client), `user.botProduct` (the identified product, when a named detector matched) and `ingest.bot.reasons` (stable reason codes). Never drops events, destinations filter via mapping. ## Installation[​](#installation "Direct link to Installation") ``` npm install @walkeros/server-transformer-bot ``` * Integrated * Bundled ``` import { startFlow } from '@walkeros/collector'; import { transformerBot } from '@walkeros/server-transformer-bot'; await startFlow({ transformers: { bot: { code: transformerBot }, }, }); ``` ``` "transformers": { "bot": { "package": "@walkeros/server-transformer-bot" } } ``` ## Configuration[​](#configuration "Direct link to Configuration") This transformer uses the standard transformer config wrapper (consent, data, env, id, ...). For the shared fields see [transformer configuration](https://www.walkeros.io/docs/transformers.md#configuration). Package-specific fields live under `config.settings` and are listed below. ## Settings[​](#settings "Direct link to Settings") | Property | Type | Description | More | | ----------------- | ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---- | | `input` | `input` | Input signal sources, resolved via getMappingValue against { event, ingest }. Each defaults to "ingest.\". Listing a name here also declares that the signal is wired in your pipeline, which is what enables the absence-based checks for its family (client hints, Fetch Metadata, Accept-Language/Encoding). "ja4" and "headerNames" are reserved and unconsumed. | | | `userAgent` | `any \| array` | | | | `ip` | `any \| array` | | | | `acceptLanguage` | `any \| array` | | | | `acceptEncoding` | `any \| array` | | | | `secFetchSite` | `any \| array` | | | | `secFetchMode` | `any \| array` | | | | `secFetchDest` | `any \| array` | | | | `secFetchUser` | `any \| array` | | | | `secChUa` | `any \| array` | | | | `secChUaMobile` | `any \| array` | | | | `secChUaPlatform` | `any \| array` | | | | `accept` | `any \| array` | | | | `contentType` | `any \| array` | | | | `referer` | `any \| array` | | | | `signatureAgent` | `any \| array` | | | | `method` | `any \| array` | | | | `ja4` | `any \| array` | | | | `headerNames` | `any \| array` | | | | `output` | `output` | Output paths for the bot annotations. | | | `botScore` | `string \| boolean` | Path for the automation likelihood (0-99, higher = more automated, null when not measured). Default: "user.botScore". Use "ingest.\*" to route to pipeline scratch instead of the event, or false to disable. | | | `botCategory` | `string \| boolean` | Path for the client category: human, suspicious, automation, search-crawler, seo-tool, monitor, link-preview, ai-agent, ai-crawler, unknown. Default: "user.botCategory". | | | `botProduct` | `string \| boolean` | Path for the identified product (e.g. "ChatGPT-User", "Googlebot"), written only when a named detector matched. Default: "user.botProduct". | | | `botReasons` | `string \| boolean` | Path for the reason-code array. Default: "ingest.bot.reasons", so the codes stay available to the pipeline without weighting the analytics payload. Codes ending in \_not\_declared report which signal families are unwired. | | | `context` | `string \| string \| object \| arr` | How the request reaches the collector. An enum literal ("beacon") pins one context for every request. Any other form is a Mapping.Value resolved per request against { event, ingest }: a dot-path string ("ingest.transport"), a {key}/{value}/{fn} object, or a fallback array tried in order (\[{key: "ingest.transport"}, {value: "beacon"}]). Wire transport truth in via the source config, e.g. express ingest map transport: {key: "query.transport"} with a ?transport= param on the collect URL. A result that is not a valid context falls back to "auto" (context-independent checks only, reported as "context\_undetermined"): scored less, never scored wrong. | | | `suspiciousAt` | `number` | Graded-layer cut between category "human" and "suspicious". Default: 25. Does not affect the deterministic scores (70 and above). | | ## Mapping[​](#mapping "Direct link to Mapping") This package does not define custom rule-level settings. For the standard rule fields (consent, condition, data, batch, name, policy) see [mapping](https://www.walkeros.io/docs/mapping.md). ## Examples ### ChatGPT-User (AI agent) A person routed an AI to fetch this page. Same score as a crawler because it is still software; the category is what lets a destination keep this traffic. Event ``` { "name": "page view", "data": { "title": "Home", "id": "/" }, "id": "ev-1700000602", "trigger": "load", "entity": "page", "action": "view", "timestamp": 1700000600, "source": { "type": "express", "platform": "server" } } ``` Out ``` return { "event": { "name": "page view", "data": { "title": "Home", "id": "/" }, "id": "ev-1700000602", "trigger": "load", "entity": "page", "action": "view", "timestamp": 1700000600, "source": { "type": "express", "platform": "server" }, "user": { "botScore": 90, "botCategory": "ai-agent", "botProduct": "ChatGPT-User" } } } ``` ### GPTBot training crawler OpenAI training crawler. The category says what it is, the product says which one. Event ``` { "name": "page view", "data": { "title": "Home", "id": "/" }, "id": "ev-1700000601", "trigger": "load", "entity": "page", "action": "view", "timestamp": 1700000600, "source": { "type": "express", "platform": "server" } } ``` Out ``` return { "event": { "name": "page view", "data": { "title": "Home", "id": "/" }, "id": "ev-1700000601", "trigger": "load", "entity": "page", "action": "view", "timestamp": 1700000600, "source": { "type": "express", "platform": "server" }, "user": { "botScore": 90, "botCategory": "ai-crawler", "botProduct": "GPTBot" } } } ``` ### Client hints contradict the UA The UA claims Chrome 124, Sec-CH-UA says Chromium 98. Graded evidence, not proof: frozen WebView UAs and enterprise UA-reduction policies produce the same mismatch, so the outcome is "suspicious" and never "automation". Event ``` { "name": "page view", "data": { "title": "Home", "id": "/" }, "id": "ev-1700000606", "trigger": "load", "entity": "page", "action": "view", "timestamp": 1700000600, "source": { "type": "express", "platform": "server" } } ``` Out ``` return { "event": { "name": "page view", "data": { "title": "Home", "id": "/" }, "id": "ev-1700000606", "trigger": "load", "entity": "page", "action": "view", "timestamp": 1700000600, "source": { "type": "express", "platform": "server" }, "user": { "botScore": 30, "botCategory": "suspicious" } } } ``` ### Human visitor (Chrome) Modern Chrome UA. No bot signals. Event ``` { "name": "page view", "data": { "title": "Home", "id": "/" }, "id": "ev-1700000600", "trigger": "load", "entity": "page", "action": "view", "timestamp": 1700000600, "source": { "type": "express", "platform": "server" } } ``` Out ``` return { "event": { "name": "page view", "data": { "title": "Home", "id": "/" }, "id": "ev-1700000600", "trigger": "load", "entity": "page", "action": "view", "timestamp": 1700000600, "source": { "type": "express", "platform": "server" }, "user": { "botScore": 0, "botCategory": "human" } } } ``` ### Wildcard Accept on an annotated pixel The pixel embed URL carries ?transport=pixel and the source lifts it into ingest, so this request is scored against the pixel profile. Browsers send a typed image Accept when they load an image, so the wildcard is worth 25. Event ``` { "name": "page view", "data": { "title": "Home", "id": "/" }, "id": "ev-1700000608", "trigger": "load", "entity": "page", "action": "view", "timestamp": 1700000600, "source": { "type": "express", "platform": "server" } } ``` Out ``` return { "event": { "name": "page view", "data": { "title": "Home", "id": "/" }, "id": "ev-1700000608", "trigger": "load", "entity": "page", "action": "view", "timestamp": 1700000600, "source": { "type": "express", "platform": "server" }, "user": { "botScore": 25, "botCategory": "suspicious" } } } ``` ### Googlebot (search crawler) Correlates with organic discoverability, so it is worth separating from both AI crawlers and unnamed automation. Event ``` { "name": "page view", "data": { "title": "Home", "id": "/" }, "id": "ev-1700000605", "trigger": "load", "entity": "page", "action": "view", "timestamp": 1700000600, "source": { "type": "express", "platform": "server" } } ``` Out ``` return { "event": { "name": "page view", "data": { "title": "Home", "id": "/" }, "id": "ev-1700000605", "trigger": "load", "entity": "page", "action": "view", "timestamp": 1700000600, "source": { "type": "express", "platform": "server" }, "user": { "botScore": 90, "botCategory": "search-crawler", "botProduct": "Googlebot" } } } ``` ### Unannotated request falls back to beacon Same instance, same headers, no ?transport= on the request, so the trailing { value: "beacon" } pins beacon. A wildcard Accept is exactly what navigator.sendBeacon sends, and the identical request now scores 0. This is what pinning per request buys: one header, two correct readings. Event ``` { "name": "page view", "data": { "title": "Home", "id": "/" }, "id": "ev-1700000609", "trigger": "load", "entity": "page", "action": "view", "timestamp": 1700000600, "source": { "type": "express", "platform": "server" } } ``` Out ``` return { "event": { "name": "page view", "data": { "title": "Home", "id": "/" }, "id": "ev-1700000609", "trigger": "load", "entity": "page", "action": "view", "timestamp": 1700000600, "source": { "type": "express", "platform": "server" }, "user": { "botScore": 0, "botCategory": "human" } } } ``` ## Source prerequisite[​](#source-prerequisite "Direct link to Source prerequisite") The transformer reads its signals from `ctx.ingest` (default path `ingest.`). The upstream server source must populate them via `config.ingest`. With nothing in ingest the score is `null` and the category is `unknown`. ``` { "sources": { "express": { "package": "@walkeros/server-source-express", "config": { "ingest": { "map": { "userAgent": { "key": "headers.user-agent" }, "acceptLanguage": { "key": "headers.accept-language" }, "acceptEncoding": { "key": "headers.accept-encoding" }, "secFetchSite": { "key": "headers.sec-fetch-site" }, "secFetchMode": { "key": "headers.sec-fetch-mode" }, "secFetchDest": { "key": "headers.sec-fetch-dest" }, "secFetchUser": { "key": "headers.sec-fetch-user" }, "secChUa": { "key": "headers.sec-ch-ua" }, "secChUaMobile": { "key": "headers.sec-ch-ua-mobile" }, "secChUaPlatform": { "key": "headers.sec-ch-ua-platform" }, "accept": { "key": "headers.accept" }, "contentType": { "key": "headers.content-type" }, "referer": { "key": "headers.referer" }, "signatureAgent": { "key": "headers.signature-agent" }, "method": { "key": "method" }, "transport": { "key": "query.transport" } } } } } } } ``` `transport` is not a signal, it is how the request context enters the pipeline. See [Request context](#request-context). Header-driven scoring works on the express and GCP Cloud Function sources. On AWS Lambda it depends on the API Gateway version's header casing, and on the `fetch` source it is unavailable: that source's raw scope is a WHATWG `Request`, whose `headers` is a `Headers` instance rather than a plain object, so dot paths resolve to `undefined`. ## Declared signals[​](#declared-signals "Direct link to Declared signals") An absence-based check runs only when its input name appears explicitly in `settings.input`. The package cannot tell "the client sent no `Sec-CH-UA`" from "the operator never mapped it", so listing a name is you asserting the signal is wired. Reading still falls back to the defaults, so listing a name with its own default path costs nothing and unlocks the check. Until then `ingest.bot.reasons` reports `ch_not_declared`, `fetchmeta_not_declared` or `accept_not_declared`, naming exactly which mapping to add. ``` { "transformers": { "bot": { "package": "@walkeros/server-transformer-bot", "config": { "settings": { "context": "beacon", "input": { "acceptLanguage": "ingest.acceptLanguage", "acceptEncoding": "ingest.acceptEncoding", "secFetchSite": "ingest.secFetchSite", "secFetchMode": "ingest.secFetchMode", "secFetchDest": "ingest.secFetchDest", "secChUa": "ingest.secChUa" } } } } } } ``` ## Request context[​](#request-context "Direct link to Request context") The same header value means opposite things in different contexts: a wildcard `Accept` is what every browser sends on a beacon and a strong bot signal on an image pixel. `settings.context` tells the context-dependent checks how the request was made. It takes three forms. **An enum literal pins one context for every request.** One of `auto`, `navigation`, `pixel`, `beacon`, `fetch` or `server`. Use it when an instance only ever sees one kind of traffic. Unset behaves like `auto`. **A per-request lookup resolves the context from the request.** Any other `Mapping.Value` is resolved per request against `{ event, ingest }`: a dot-path string such as `"ingest.transport"`, or a `{ "key": … }` / `{ "value": … }` / `{ "fn": … }` object. **A fallback array tries its entries in order.** `[{ "key": "ingest.transport" }, { "value": "beacon" }]` uses the sender's annotation when there is one and pins `beacon` otherwise. A bare string is always read as a lookup path, so a string context that is not one of the six literals must contain a dot. A typo like `"beacn"` fails validation instead of silently becoming a lookup that resolves to nothing. The guard applies inside the array too: a literal there is written `{ "value": "beacon" }`, and the bare `["beacon"]` is rejected. ### Wiring the transport[​](#wiring-the-transport "Direct link to Wiring the transport") The sender declares the transport, the source lifts it into `ingest`, the transformer resolves it per request. Neither package knows the other exists, and one instance then serves a deployment that receives beacons, pixels and fetches on the same endpoint. 1. Add `?transport=beacon` to the collect URL your web destination posts to, and `?transport=pixel` to the pixel embed URL. 2. Map it in the source's ingest block with `"transport": { "key": "query.transport" }`. The normalized request scope already exposes the parsed query, so nothing else is needed. 3. Point the bot config at `ingest.transport`, with a static entry behind it for senders that carry no annotation. ``` ``` ``` { "sources": { "express": { "package": "@walkeros/server-source-express", "config": { "ingest": { "map": { "userAgent": { "key": "headers.user-agent" }, "accept": { "key": "headers.accept" }, "transport": { "key": "query.transport" } } } } } }, "transformers": { "bot": { "package": "@walkeros/server-transformer-bot", "config": { "settings": { "context": [{ "key": "ingest.transport" }, { "value": "beacon" }] } } } } } ``` A query parameter is a claim by whoever controls the sender. That is exactly right for telling your own transports apart, and worthless against a client that wants to be scored as a beacon. For server truth instead, give pixels their own route and derive the context from `ingest.path` with a `fn`. ### When resolution fails[​](#when-resolution-fails "Direct link to When resolution fails") Anything that resolves outside the vocabulary, and an unannotated request whose chain has no static entry, both land on `auto`. `auto` never determines a context: absence of `Sec-Fetch-*` is both a signal worth scoring and the reason auto-derivation fails, and the request method does not rescue it. So `auto` runs the context-independent checks only and reports `context_undetermined` in `ingest.bot.reasons`. The failure mode is "we scored less", never "we scored wrong". Pinning, however it is reached, unlocks the `Accept` shape check, the Fetch Metadata profile comparison and the beacon `Content-Type` check. `server` means server-to-server ingestion and disables every browser-shaped check. The client-hint coherence family, the most discriminating header signal available, is context-independent and works everywhere. ## Detection layers[​](#detection-layers "Direct link to Detection layers") Deterministic, most specific first, first match wins. The graded layer does not run when one of these fires. | Score | Category | Trigger | | ----- | ------------ | ----------------------------------------- | | 70 | `automation` | missing User-Agent | | 90 | named | AI agent or AI crawler UA map | | 90 | named | non-AI crawler UA map | | 80 | `automation` | `isbot` | | 75 | `automation` | a value impossible for the pinned context | Graded, capped at 60, so the layers never overlap: `ch_version_mismatch` (30), `ch_missing_on_chromium` (25), `accept_generic_on_typed_context` (25), `fetchmeta_missing_on_modern_ua` (15), `fetchmeta_profile_mismatch` (15), `accept_language_missing` (10), `accept_encoding_missing` (5). The weights and the `suspiciousAt` default of 25 are provisional starting values, not calibrated against a labelled corpus. ## Categories[​](#categories "Direct link to Categories") | Value | Meaning | Score | | ---------------- | ---------------------------------------------------------------- | ------ | | `human` | no evidence, or graded sum below the cut | 0-24 | | `suspicious` | graded evidence, nothing decisive | 25-60 | | `automation` | isbot, missing UA, or impossible-for-context values | 70-80 | | `search-crawler` | search engine index; correlates with organic discoverability | 90 | | `seo-tool` | third-party commercial crawler | 90 | | `monitor` | uptime and synthetic monitoring, usually your own infrastructure | 90 | | `link-preview` | link unfurler, meaning a person just shared this URL | 90 | | `ai-agent` | AI agent acting for a person | 90 | | `ai-crawler` | AI training or search-index crawler | 90 | | `unknown` | nothing resolvable at all | `null` | Which categories can appear depends on how events reach the pipeline. Link unfurlers do not run JavaScript, so `link-preview` is structurally invisible to a JS tag and real only for server-side collection. Googlebot renders, so `search-crawler` does fire in a JS tag. The README carries the full reachability table. ## Destination filtering recipes[​](#destination-filtering-recipes "Direct link to Destination filtering recipes") Drop everything automated: `event.user.botScore > 50` Drop crawlers, keep the AI traffic a person triggered: `event.user.botScore > 50 AND event.user.botCategory != 'ai-agent'` Keep link unfurls, which mean somebody just shared the URL: `event.user.botCategory != 'link-preview'` AI visibility report: `event.user.botCategory IN ('ai-agent', 'ai-crawler')`, grouped by `event.user.botProduct` `botScore` is `null` for category `unknown`, which means not measured rather than human, and evaluates false under a `> 50` filter. Cloudflare's `bot_score` runs the opposite way (1 = bot) because it is a trust score; ours matches its field name. ## Not yet implemented[​](#not-yet-implemented "Direct link to Not yet implemented") Identity verification (`settings.verify`, a CIDR matcher and a `botVerified` output), Web Bot Auth signature verification, reverse DNS, ASN / datacenter-IP, web-side runtime checks, behavioral signals, and TLS / JA4. The input names `ja4` and `headerNames` are reserved and resolved but unconsumed. See the README for why each one is out. ## Limits[​](#limits "Direct link to Limits") Will not catch residential-proxy + stealth Chrome, CAPTCHA-solver farms, or real-browser-as-a-service. For that threat model use a commercial vendor (Cloudflare Bot Management, DataDome, HUMAN). In-browser agents such as Claude for Chrome are undetectable here by construction: they drive a real browser session and produce headers identical to it. A UA-map match is a claim, not proof: Screaming Frog ships Googlebot and Bingbot presets, and any client can send any UA.