Bot detection

Server Source code Package

Annotates events with user.botScore (0-99, higher = more automated), user.botCategory (what kind of client), user.botProduct (the identified product, when a named detector matched) and ingest.bot.reasons (stable reason codes). Never drops events, destinations filter via mapping.

Installation

npm install @walkeros/server-transformer-bot

Integrated
Bundled

import { startFlow } from '@walkeros/collector';
import { transformerBot } from '@walkeros/server-transformer-bot';

await startFlow({
transformers: {
bot: { code: transformerBot },
},
});

"transformers": {
  "bot": {
    "package": "@walkeros/server-transformer-bot"
  }
}

Configuration

This transformer uses the standard transformer config wrapper (consent, data, env, id, ...). For the shared fields see transformer configuration. Package-specific fields live under config.settings and are listed below.

Settings

Property	Type	Description
`input`	`input`	Input signal sources, resolved via getMappingValue against { event, ingest }. Each defaults to "ingest.<name>". Listing a name here also declares that the signal is wired in your pipeline, which is what enables the absence-based checks for its family (client hints, Fetch Metadata, Accept-Language/Encoding). "ja4" and "headerNames" are reserved and unconsumed.
`userAgent`	`any \| array`
`ip`	`any \| array`
`acceptLanguage`	`any \| array`
`acceptEncoding`	`any \| array`
`secFetchSite`	`any \| array`
`secFetchMode`	`any \| array`
`secFetchDest`	`any \| array`
`secFetchUser`	`any \| array`
`secChUa`	`any \| array`
`secChUaMobile`	`any \| array`
`secChUaPlatform`	`any \| array`
`accept`	`any \| array`
`contentType`	`any \| array`
`referer`	`any \| array`
`signatureAgent`	`any \| array`
`method`	`any \| array`
`ja4`	`any \| array`
`headerNames`	`any \| array`
`output`	`output`	Output paths for the bot annotations.
`botScore`	`string \| boolean`	Path for the automation likelihood (0-99, higher = more automated, null when not measured). Default: "user.botScore". Use "ingest.*" to route to pipeline scratch instead of the event, or false to disable.
`botCategory`	`string \| boolean`	Path for the client category: human, suspicious, automation, search-crawler, seo-tool, monitor, link-preview, ai-agent, ai-crawler, unknown. Default: "user.botCategory".
`botProduct`	`string \| boolean`	Path for the identified product (e.g. "ChatGPT-User", "Googlebot"), written only when a named detector matched. Default: "user.botProduct".
`botReasons`	`string \| boolean`	Path for the reason-code array. Default: "ingest.bot.reasons", so the codes stay available to the pipeline without weighting the analytics payload. Codes ending in _not_declared report which signal families are unwired.
`context`		How the request reaches the collector. An enum literal ("beacon") pins one context for every request. Any other form is a Mapping.Value resolved per request against { event, ingest }: a dot-path string ("ingest.transport"), a {key}/{value}/{fn} object, or a fallback array tried in order ([{key: "ingest.transport"}, {value: "beacon"}]). Wire transport truth in via the source config, e.g. express ingest map transport: {key: "query.transport"} with a ?transport= param on the collect URL. A result that is not a valid context falls back to "auto" (context-independent checks only, reported as "context_undetermined"): scored less, never scored wrong.
`suspiciousAt`	`number`	Graded-layer cut between category "human" and "suspicious". Default: 25. Does not affect the deterministic scores (70 and above).

Mapping

This package does not define custom rule-level settings. For the standard rule fields (consent, condition, data, batch, name, policy) see mapping.

Examples

ChatGPT-User (AI agent)

A person routed an AI to fetch this page. Same score as a crawler because it is still software; the category is what lets a destination keep this traffic.

Event

{
  "name": "page view",
  "data": {
    "title": "Home",
    "id": "/"
  },
  "id": "ev-1700000602",
  "trigger": "load",
  "entity": "page",
  "action": "view",
  "timestamp": 1700000600,
  "source": {
    "type": "express",
    "platform": "server"
  }
}

Out

return {
  "event": {
    "name": "page view",
    "data": {
      "title": "Home",
      "id": "/"
    },
    "id": "ev-1700000602",
    "trigger": "load",
    "entity": "page",
    "action": "view",
    "timestamp": 1700000600,
    "source": {
      "type": "express",
      "platform": "server"
    },
    "user": {
      "botScore": 90,
      "botCategory": "ai-agent",
      "botProduct": "ChatGPT-User"
    }
  }
}

GPTBot training crawler

OpenAI training crawler. The category says what it is, the product says which one.

Event

{
  "name": "page view",
  "data": {
    "title": "Home",
    "id": "/"
  },
  "id": "ev-1700000601",
  "trigger": "load",
  "entity": "page",
  "action": "view",
  "timestamp": 1700000600,
  "source": {
    "type": "express",
    "platform": "server"
  }
}

Out

return {
  "event": {
    "name": "page view",
    "data": {
      "title": "Home",
      "id": "/"
    },
    "id": "ev-1700000601",
    "trigger": "load",
    "entity": "page",
    "action": "view",
    "timestamp": 1700000600,
    "source": {
      "type": "express",
      "platform": "server"
    },
    "user": {
      "botScore": 90,
      "botCategory": "ai-crawler",
      "botProduct": "GPTBot"
    }
  }
}

Client hints contradict the UA

The UA claims Chrome 124, Sec-CH-UA says Chromium 98. Graded evidence, not proof: frozen WebView UAs and enterprise UA-reduction policies produce the same mismatch, so the outcome is "suspicious" and never "automation".

Event

{
  "name": "page view",
  "data": {
    "title": "Home",
    "id": "/"
  },
  "id": "ev-1700000606",
  "trigger": "load",
  "entity": "page",
  "action": "view",
  "timestamp": 1700000600,
  "source": {
    "type": "express",
    "platform": "server"
  }
}

Out

return {
  "event": {
    "name": "page view",
    "data": {
      "title": "Home",
      "id": "/"
    },
    "id": "ev-1700000606",
    "trigger": "load",
    "entity": "page",
    "action": "view",
    "timestamp": 1700000600,
    "source": {
      "type": "express",
      "platform": "server"
    },
    "user": {
      "botScore": 30,
      "botCategory": "suspicious"
    }
  }
}

Human visitor (Chrome)

Modern Chrome UA. No bot signals.

Event

{
  "name": "page view",
  "data": {
    "title": "Home",
    "id": "/"
  },
  "id": "ev-1700000600",
  "trigger": "load",
  "entity": "page",
  "action": "view",
  "timestamp": 1700000600,
  "source": {
    "type": "express",
    "platform": "server"
  }
}

Out

return {
  "event": {
    "name": "page view",
    "data": {
      "title": "Home",
      "id": "/"
    },
    "id": "ev-1700000600",
    "trigger": "load",
    "entity": "page",
    "action": "view",
    "timestamp": 1700000600,
    "source": {
      "type": "express",
      "platform": "server"
    },
    "user": {
      "botScore": 0,
      "botCategory": "human"
    }
  }
}

Wildcard Accept on an annotated pixel

The pixel embed URL carries ?transport=pixel and the source lifts it into ingest, so this request is scored against the pixel profile. Browsers send a typed image Accept when they load an image, so the wildcard is worth 25.

Event

{
  "name": "page view",
  "data": {
    "title": "Home",
    "id": "/"
  },
  "id": "ev-1700000608",
  "trigger": "load",
  "entity": "page",
  "action": "view",
  "timestamp": 1700000600,
  "source": {
    "type": "express",
    "platform": "server"
  }
}

Out

return {
  "event": {
    "name": "page view",
    "data": {
      "title": "Home",
      "id": "/"
    },
    "id": "ev-1700000608",
    "trigger": "load",
    "entity": "page",
    "action": "view",
    "timestamp": 1700000600,
    "source": {
      "type": "express",
      "platform": "server"
    },
    "user": {
      "botScore": 25,
      "botCategory": "suspicious"
    }
  }
}

Googlebot (search crawler)

Correlates with organic discoverability, so it is worth separating from both AI crawlers and unnamed automation.

Event

{
  "name": "page view",
  "data": {
    "title": "Home",
    "id": "/"
  },
  "id": "ev-1700000605",
  "trigger": "load",
  "entity": "page",
  "action": "view",
  "timestamp": 1700000600,
  "source": {
    "type": "express",
    "platform": "server"
  }
}

Out

return {
  "event": {
    "name": "page view",
    "data": {
      "title": "Home",
      "id": "/"
    },
    "id": "ev-1700000605",
    "trigger": "load",
    "entity": "page",
    "action": "view",
    "timestamp": 1700000600,
    "source": {
      "type": "express",
      "platform": "server"
    },
    "user": {
      "botScore": 90,
      "botCategory": "search-crawler",
      "botProduct": "Googlebot"
    }
  }
}

Unannotated request falls back to beacon

Same instance, same headers, no ?transport= on the request, so the trailing { value: "beacon" } pins beacon. A wildcard Accept is exactly what navigator.sendBeacon sends, and the identical request now scores 0. This is what pinning per request buys: one header, two correct readings.

Event

{
  "name": "page view",
  "data": {
    "title": "Home",
    "id": "/"
  },
  "id": "ev-1700000609",
  "trigger": "load",
  "entity": "page",
  "action": "view",
  "timestamp": 1700000600,
  "source": {
    "type": "express",
    "platform": "server"
  }
}

Out

return {
  "event": {
    "name": "page view",
    "data": {
      "title": "Home",
      "id": "/"
    },
    "id": "ev-1700000609",
    "trigger": "load",
    "entity": "page",
    "action": "view",
    "timestamp": 1700000600,
    "source": {
      "type": "express",
      "platform": "server"
    },
    "user": {
      "botScore": 0,
      "botCategory": "human"
    }
  }
}

Source prerequisite

The transformer reads its signals from ctx.ingest (default path ingest.<name>). The upstream server source must populate them via config.ingest. With nothing in ingest the score is null and the category is unknown.

{
  "sources": {
    "express": {
      "package": "@walkeros/server-source-express",
      "config": {
        "ingest": {
          "map": {
            "userAgent": { "key": "headers.user-agent" },
            "acceptLanguage": { "key": "headers.accept-language" },
            "acceptEncoding": { "key": "headers.accept-encoding" },
            "secFetchSite": { "key": "headers.sec-fetch-site" },
            "secFetchMode": { "key": "headers.sec-fetch-mode" },
            "secFetchDest": { "key": "headers.sec-fetch-dest" },
            "secFetchUser": { "key": "headers.sec-fetch-user" },
            "secChUa": { "key": "headers.sec-ch-ua" },
            "secChUaMobile": { "key": "headers.sec-ch-ua-mobile" },
            "secChUaPlatform": { "key": "headers.sec-ch-ua-platform" },
            "accept": { "key": "headers.accept" },
            "contentType": { "key": "headers.content-type" },
            "referer": { "key": "headers.referer" },
            "signatureAgent": { "key": "headers.signature-agent" },
            "method": { "key": "method" },
            "transport": { "key": "query.transport" }
          }
        }
      }
    }
  }
}

transport is not a signal, it is how the request context enters the pipeline. See Request context.

Header-driven scoring works on the express and GCP Cloud Function sources. On AWS Lambda it depends on the API Gateway version's header casing, and on the fetch source it is unavailable: that source's raw scope is a WHATWG Request, whose headers is a Headers instance rather than a plain object, so dot paths resolve to undefined.

Declared signals

An absence-based check runs only when its input name appears explicitly in settings.input. The package cannot tell "the client sent no Sec-CH-UA" from "the operator never mapped it", so listing a name is you asserting the signal is wired. Reading still falls back to the defaults, so listing a name with its own default path costs nothing and unlocks the check.

Until then ingest.bot.reasons reports ch_not_declared, fetchmeta_not_declared or accept_not_declared, naming exactly which mapping to add.

{
  "transformers": {
    "bot": {
      "package": "@walkeros/server-transformer-bot",
      "config": {
        "settings": {
          "context": "beacon",
          "input": {
            "acceptLanguage": "ingest.acceptLanguage",
            "acceptEncoding": "ingest.acceptEncoding",
            "secFetchSite": "ingest.secFetchSite",
            "secFetchMode": "ingest.secFetchMode",
            "secFetchDest": "ingest.secFetchDest",
            "secChUa": "ingest.secChUa"
          }
        }
      }
    }
  }
}

Request context

The same header value means opposite things in different contexts: a wildcard Accept is what every browser sends on a beacon and a strong bot signal on an image pixel. settings.context tells the context-dependent checks how the request was made. It takes three forms.

An enum literal pins one context for every request. One of auto, navigation, pixel, beacon, fetch or server. Use it when an instance only ever sees one kind of traffic. Unset behaves like auto.

A per-request lookup resolves the context from the request. Any other Mapping.Value is resolved per request against { event, ingest }: a dot-path string such as "ingest.transport", or a { "key": … } / { "value": … } / { "fn": … } object.

A fallback array tries its entries in order. [{ "key": "ingest.transport" }, { "value": "beacon" }] uses the sender's annotation when there is one and pins beacon otherwise.

A bare string is always read as a lookup path, so a string context that is not one of the six literals must contain a dot. A typo like "beacn" fails validation instead of silently becoming a lookup that resolves to nothing. The guard applies inside the array too: a literal there is written { "value": "beacon" }, and the bare ["beacon"] is rejected.

Wiring the transport

The sender declares the transport, the source lifts it into ingest, the transformer resolves it per request. Neither package knows the other exists, and one instance then serves a deployment that receives beacons, pixels and fetches on the same endpoint.

Add ?transport=beacon to the collect URL your web destination posts to, and ?transport=pixel to the pixel embed URL.
Map it in the source's ingest block with "transport": { "key": "query.transport" }. The normalized request scope already exposes the parsed query, so nothing else is needed.
Point the bot config at ingest.transport, with a static entry behind it for senders that carry no annotation.

<img src="https://collect.example.com/px.gif?transport=pixel" width="1" height="1" alt="">

{
  "sources": {
    "express": {
      "package": "@walkeros/server-source-express",
      "config": {
        "ingest": {
          "map": {
            "userAgent": { "key": "headers.user-agent" },
            "accept": { "key": "headers.accept" },
            "transport": { "key": "query.transport" }
          }
        }
      }
    }
  },
  "transformers": {
    "bot": {
      "package": "@walkeros/server-transformer-bot",
      "config": {
        "settings": {
          "context": [{ "key": "ingest.transport" }, { "value": "beacon" }]
        }
      }
    }
  }
}

A query parameter is a claim by whoever controls the sender. That is exactly right for telling your own transports apart, and worthless against a client that wants to be scored as a beacon. For server truth instead, give pixels their own route and derive the context from ingest.path with a fn.

When resolution fails

Anything that resolves outside the vocabulary, and an unannotated request whose chain has no static entry, both land on auto. auto never determines a context: absence of Sec-Fetch-* is both a signal worth scoring and the reason auto-derivation fails, and the request method does not rescue it. So auto runs the context-independent checks only and reports context_undetermined in ingest.bot.reasons. The failure mode is "we scored less", never "we scored wrong".

Pinning, however it is reached, unlocks the Accept shape check, the Fetch Metadata profile comparison and the beacon Content-Type check. server means server-to-server ingestion and disables every browser-shaped check. The client-hint coherence family, the most discriminating header signal available, is context-independent and works everywhere.

Detection layers

Deterministic, most specific first, first match wins. The graded layer does not run when one of these fires.

Score	Category	Trigger
70	`automation`	missing User-Agent
90	named	AI agent or AI crawler UA map
90	named	non-AI crawler UA map
80	`automation`	`isbot`
75	`automation`	a value impossible for the pinned context

Graded, capped at 60, so the layers never overlap: ch_version_mismatch (30), ch_missing_on_chromium (25), accept_generic_on_typed_context (25), fetchmeta_missing_on_modern_ua (15), fetchmeta_profile_mismatch (15), accept_language_missing (10), accept_encoding_missing (5). The weights and the suspiciousAt default of 25 are provisional starting values, not calibrated against a labelled corpus.

Value	Meaning	Score
`human`	no evidence, or graded sum below the cut	0-24
`suspicious`	graded evidence, nothing decisive	25-60
`automation`	isbot, missing UA, or impossible-for-context values	70-80
`search-crawler`	search engine index; correlates with organic discoverability	90
`seo-tool`	third-party commercial crawler	90
`monitor`	uptime and synthetic monitoring, usually your own infrastructure	90
`link-preview`	link unfurler, meaning a person just shared this URL	90
`ai-agent`	AI agent acting for a person	90
`ai-crawler`	AI training or search-index crawler	90
`unknown`	nothing resolvable at all	`null`

Destination filtering recipes

Drop everything automated: event.user.botScore > 50

Drop crawlers, keep the AI traffic a person triggered: event.user.botScore > 50 AND event.user.botCategory != 'ai-agent'

Keep link unfurls, which mean somebody just shared the URL: event.user.botCategory != 'link-preview'

AI visibility report: event.user.botCategory IN ('ai-agent', 'ai-crawler'), grouped by event.user.botProduct

botScore is null for category unknown, which means not measured rather than human, and evaluates false under a > 50 filter. Cloudflare's bot_score runs the opposite way (1 = bot) because it is a trust score; ours matches its field name.

Not yet implemented

Identity verification (settings.verify, a CIDR matcher and a botVerified output), Web Bot Auth signature verification, reverse DNS, ASN / datacenter-IP, web-side runtime checks, behavioral signals, and TLS / JA4. The input names ja4 and headerNames are reserved and resolved but unconsumed. See the README for why each one is out.

Limits

Will not catch residential-proxy + stealth Chrome, CAPTCHA-solver farms, or real-browser-as-a-service. For that threat model use a commercial vendor (Cloudflare Bot Management, DataDome, HUMAN).

In-browser agents such as Claude for Chrome are undetectable here by construction: they drive a real browser session and produce headers identical to it. A UA-map match is a claim, not proof: Screaming Frog ships Googlebot and Bingbot presets, and any client can send any UA.

💡 Need implementation support?

elbwalker offers hands-on support: setup review, measurement planning, destination mapping, and live troubleshooting. Book a 2-hour session (€399)

Installation​

Configuration​

Settings​

Mapping​

Examples

ChatGPT-User (AI agent)

GPTBot training crawler

Client hints contradict the UA

Human visitor (Chrome)

Wildcard Accept on an annotated pixel

Googlebot (search crawler)

Unannotated request falls back to beacon

Source prerequisite​

Declared signals​

Request context​

Wiring the transport​

When resolution fails​

Detection layers​

Categories​

Destination filtering recipes​

Not yet implemented​

Limits​

Installation

Configuration

Settings

Mapping

Source prerequisite

Declared signals

Request context

Wiring the transport

When resolution fails

Detection layers

Categories

Destination filtering recipes

Not yet implemented

Limits