Web Bot Auth: Cryptographic Identity for Agents on HTTP

The agentic web has a payment rail (x402) and a tool protocol (MCP). What it did not have, until recently, was a way for an agent to prove who it is to a server it has never met. Web Bot Auth — HTTP Message Signatures with a published key directory — is becoming that layer, and in 2026 it moved from a Cloudflare experiment to an IETF working group with production deployments at two of the largest edge networks.

This matters to anyone operating autonomous agents. The web is closing to anonymous automation: origins are blocking unidentified bots by default, and the infrastructure that lets a verified agent through is the same infrastructure that will charge it per request. Identity and payment are converging into a single handshake. This post walks the wire format, the standards track, the deployments, and the exact point where a signed request turns into a paid one.

Why the old identifiers failed

Servers historically had two ways to decide whether automated traffic was welcome: the User-Agent header and the source IP address. Both are broken for agents, and Cloudflare's original Web Bot Auth proposal (May 2025) is blunt about why.

A User-Agent string is a self-reported claim with no verification mechanism. Anyone can send User-Agent: MyTrustedAgent/1.0. IP allowlists are stronger but brittle: agents increasingly run from shared cloud infrastructure, behind privacy proxies and VPNs, where an IP range says which datacenter a request left from, not which operator controls the software. Ranges also change constantly, so every origin ends up maintaining ad-hoc CIDR lists that are stale the day they ship. Shared secrets do not scale either — a crawler cannot provision a separate token with every website on the internet before visiting it.

The failure is structural. All three mechanisms tie identity to where a request comes from or what it claims to be. None of them tie identity to something only the operator possesses. That is a job for asymmetric cryptography.

RFC 9421: signatures over HTTP messages

RFC 9421, HTTP Message Signatures, is the IETF standard for creating and verifying signatures over selected components of an HTTP message — specific headers, plus derived components like @authority (the target host) and @target-uri. The signer declares exactly which components it covered and under which parameters; the verifier reconstructs the same signature base and checks the signature against a known public key.

RFC 9421 is deliberately generic. It does not say who signs, where keys live, or what a signature means. Web Bot Auth is a profile of RFC 9421 that answers those three questions for automated traffic: the bot's provider signs every request, keys are published at a well-known HTTPS location, and a valid signature means "this request was produced by the operator that controls this key directory."

The wire format

A Web Bot Auth request carries up to three extra headers. Per Cloudflare's implementation docs and the current protocol draft, they look like this:

# Where the verifier can find the public keys
Signature-Agent: "https://signer.example.com"

# What was signed, and under which parameters
Signature-Input: sig=("@authority" "signature-agent");\
                 created=1700000000;\
                 expires=1700011111;\
                 keyid="ba3e64==";\
                 tag="web-bot-auth"

# The Ed25519 signature itself, base64-encoded
Signature: sig=abc==

Each parameter does real work. created and expires bound the validity window — the current draft recommends signatures live no more than 24 hours, which limits replay. keyid is not a free-form string: it must be the base64url JWK SHA-256 thumbprint (RFC 7638) of the signing key, so the verifier can locate the exact key in the directory without ambiguity. tag must equal web-bot-auth, scoping the signature to this protocol so it cannot be confused with a signature made for another purpose. And the signed components must include the target — at least @authority in Cloudflare's implementation; at least one of @authority or @target-uri in the current draft — so a signature captured on one site cannot be replayed against another.

The verifier side hinges on the key directory. An operator publishes a JSON Web Key Set at a fixed path on the domain named in Signature-Agent:

GET https://signer.example.com/.well-known/http-message-signatures-directory

{
  "keys": [{
    "kty": "OKP",
    "crv": "Ed25519",
    "x": "JrQLj5P_89iXES9-vFgrIy29clF9CC_oPPsw3c5D0bs"
  }]
}

Ed25519 is the only algorithm Cloudflare supports — one curve, no negotiation, no downgrade surface. The directory must be served over HTTPS, and Cloudflare's verifier additionally validates a signature over the directory response itself before trusting the keys inside it. The full loop: fetch the directory, verify its own message signature, extract the Ed25519 keys, match the request's keyid thumbprint, verify the request signature. No IP lists anywhere in the path.

From individual drafts to an IETF working group

The standards trajectory is worth tracking precisely, because it tells you how much to build on this today. Web Bot Auth started as individual drafts by Thibault Meunier (Cloudflare): an architecture document and a key-directory document. The architecture draft, draft-meunier-web-bot-auth-architecture, reached version 05 in March 2026 and has since been replaced by draft-meunier-webbotauth-httpsig-protocol ("HTTP Message Signatures for automated traffic"), whose -00 landed in June 2026 with authors from Cloudflare and Google.

More significant: the IETF chartered a dedicated Web Bot Auth working group (webbotauth) in the Web and Internet Transport area, chaired by David Schinazi and Rifaat Shekh-Yusef. Its charter milestones target standards-track specifications for the authentication technique and for conveying additional bot metadata, plus a Best Current Practice operational document by the end of August 2026. Individual drafts becoming a chartered WG is the difference between "Cloudflare's idea" and "the way the web will do this" — with the usual caveat that nothing is an RFC yet and the drafts explicitly carry no formal standing until the WG ships them.

Discovery is being standardized in parallel. A signature tells you the keys live at signer.example.com, but it does not tell an origin which signature agents exist or what they do. Cloudflare's registry proposal (October 2025, draft-meunier-webbotauth-registry) defines a lightweight format: lists of signature-agent URLs that anyone can curate and host, plus "signature agent cards" that extend the JWKS directory with metadata — operator name and contact, expected user-agent, purpose, rate-control expectations, robots.txt (RFC 9309) compliance. It is the same shape as the reputation problem ERC-8004 attacks on-chain: identity is step one, discoverable reputation is step two.

Who ships it today

This is no longer a research demo. Three production deployments matter.

Cloudflare. Message signatures are a first-class verification method in the Verified Bots program: an operator generates keys, hosts the directory, signs requests, and registers the directory URL through the dashboard. Once verified, every site behind Cloudflare can act on it — the cf.verified_bot_category field is available in WAF Custom Rules, rate limiting, and Transform Rules, so an origin can express "allow academic research agents, charge AI crawlers, block everything unsigned" as ordinary rule expressions evaluated at the edge.

OpenAI. Quoted directly in Cloudflare's announcement: "With HTTP Message Signatures (RFC 9421), OpenAI signs all Operator requests so site owners can verify they genuinely originate from Operator." The browsing agent — the hardest bot to distinguish from a human — is precisely where self-identification carries the most value.

AWS. Amazon Bedrock AgentCore Browser added Web Bot Auth in preview (announced October 2025): the managed browser signs every outgoing HTTP request automatically, and bot-control vendors — Cloudflare, Akamai, HUMAN Security — verify the signatures and apply the site owner's policy, reducing CAPTCHA friction for verified agents. On the receiving side, AWS WAF Bot Control verifies Web Bot Auth signatures. Notably, AgentCore currently signs with a service-level key and intends to move to customer-specific keys as the protocol matures — the granularity question (provider identity vs. per-tenant identity) is still open.

Tooling is open. The cloudflare/web-bot-auth repo (Apache 2.0) ships TypeScript packages (web-bot-auth, http-message-sig, jsonwebkey-thumbprint) and Rust crates (web-bot-auth, http-signature-directory), plus working examples: a signing browser extension, a verifying Cloudflare Worker, and a Caddy plugin. The repo states plainly that the code is unaudited — fine for adopting the wire format, worth remembering before it guards anything valuable.

Where identity meets payment

Here is why this is not just a bot-management story. Every recent machine-payment design assumes the payer is identified by a Web Bot Auth signature before money moves.

Cloudflare's pay per crawl (July 2025) made the dependency explicit. A crawler must register its key directory and sign every request with the three Web Bot Auth headers. When it hits paid content, the edge answers 402 Payment Required with a crawler-price header; the crawler retries with crawler-exact-price to accept, or sends crawler-max-price upfront to skip the round trip; a successful response confirms the charge with crawler-charged. The signature is not an add-on — it is the billing identity. Cloudflare acts as merchant of record and settles aggregated charges to publishers against the registered, verified crawler.

The x402 "deferred" scheme Cloudflare proposed alongside the x402 Foundation pushes the same idea into the open protocol: it decouples cryptographic commitment from settlement, so an agent can accumulate signed payment commitments — authenticated by Signature-Agent and Signature-Input headers — and settle in batch daily, or under a pre-negotiated licensing agreement. Compare that with the immediate-settlement path we covered in EIP-3009, where each request carries a self-contained transferWithAuthorization: deferred trades atomicity for lower per-request overhead, and it only works because the payer has a persistent, verifiable identity to bill against.

The Monetization Gateway (waitlist opened July 1, 2026) generalizes this to everything behind Cloudflare: web pages, datasets, APIs, MCP tools. Sellers write pricing rules as edge expressions; buyers get a 402 with price, asset, and destination; a facilitator verifies payment in stablecoins (USDC among them) and the edge releases the resource — enforcement at 330+ locations before traffic ever reaches the origin. And in Cloudflare's own framing, sellers can "require agents to authenticate with Web Bot Auth and apply usage-based pricing against accounts they already hold." Identity-gated, usage-priced access is becoming a checkbox on the world's largest reverse proxy.

Put together, the stack looks like this: RFC 9421 answers who is calling, x402 answers how it pays, and the edge enforces both in a single 402 negotiation. That is the same layering we described from the payment side in our x402 deep dive, now with the missing identity layer filled in.

The honest caveats

Three limits worth stating. First, maturity: the protocol documents are drafts, the previous architecture draft is formally expired, and implementations warn that details may change before the WG produces RFCs. Building against Cloudflare's and AWS's deployed behavior is reasonable; treating the drafts as frozen is not. Second, granularity: today's signatures identify providers, not agents. Cloudflare verifying "this is AgentCore traffic" says nothing about which of thousands of tenants is driving the browser — customer-specific keys are on the roadmap, not in production. Third, gatekeeping: verification lists and registries are curated by the same edge networks that sell bot management. The registry draft's open, anyone-can-host format is the counterweight, but the default trust anchors today are Cloudflare's and AWS's databases. The x402 Foundation's move to the Linux Foundation shows the ecosystem understands this concern on the payment side; identity will need the same treatment.

What it means for LLM4Agents

LLM4Agents operates on both sides of this handshake, so Web Bot Auth cuts twice.

On the outbound side, agents funded through our gateway increasingly reach for the open web — fetching pages, calling third-party APIs, walking up to x402 endpoints. As origins default to blocking unsigned automation, an unsigned agent fleet degrades: more CAPTCHAs, more 403s, more silent failures that look like model errors but are actually access denials. Request signing becomes table stakes for reliability, the same way fallback chains are for model availability. A gateway that signs egress traffic with a published key directory gives every agent behind it a verifiable identity without each operator standing up their own signing infrastructure.

On the inbound side, our API endpoints and MCP tools receive automated traffic by design — that is the whole product. Web Bot Auth gives us a cryptographic signal to segment that traffic: verified agents from known providers can get higher rate limits or walk-up x402 pricing, unsigned traffic gets the conservative path. And the deferred-settlement pattern maps directly onto our reserve–proxy–settle billing loop: a persistent verified identity is exactly what makes post-paid or batch-settled billing safe to offer. Identity is not adjacent to the payment stack we have been building — it is the prerequisite for its next iteration.

Staying on the frontier

Concrete steps, in order.

1. Sign gateway egress. Generate an Ed25519 keypair, publish the JWKS at /.well-known/http-message-signatures-directory on the llm4agents domain, and sign outbound HTTP from agent workloads with tag="web-bot-auth", short expiries, and @authority coverage. The Apache-2.0 TypeScript and Rust libraries cover the mechanics; this is days of work, not months.

2. Register with the verifiers that matter. Submit the key directory to Cloudflare's Verified Bots program and track AWS WAF Bot Control's registration path as it opens beyond AgentCore. Being in the default-trusted set before origins tighten defaults is cheap now and expensive later.

3. Verify inbound signatures. Add RFC 9421 verification at our edge for the web-bot-auth tag, and expose the result to billing and rate-limiting policy. Start observational — log which callers already sign — then let verified identity unlock better pricing tiers.

4. Publish a signature agent card. When the registry draft stabilizes, ship the extended directory metadata: operator contact, purpose, rate expectations. Low cost, and it positions the platform for whatever reputation layer wins — registry lists on the web2 side, ERC-8004 on-chain.

5. Track the webbotauth WG and the deferred x402 scheme. The WG's output through August 2026 will fix the protocol details; the deferred scheme decides whether per-tenant keys and batch settlement become standard. Both directly shape our roadmap for per-agent identity — the granularity the ecosystem has not solved yet, and where a gateway that already knows each agent's wallet and history has a natural head start.