AIPREF at the deadline: auditing the vocabulary agents are supposed to read

Two IETF drafts are due at the IESG on 31 August 2026. One of them warns on its cover that its contents do not reflect working group consensus. We read every revision, implemented the resolution algorithm, and pointed it at 10,000 domains.

Before an agent pays for anything, it fetches something. Our previous audits covered what happens after the fetch is priced: the x402 handshake, the settlement rail, the wallet caps. This one covers the layer above it. Whether the publisher wanted the fetch at all, and whether that wish is written in a form any machine can parse.

The IETF's answer to that question is the AI Preferences working group, AIPREF. It was chartered on 9 April 2025 in the Web and Internet Transport area, chaired by Mark Nottingham and Suresh Krishnan. Its two deliverables are a vocabulary for expressing preferences and a mechanism for attaching them to content. Both carry a milestone date of 31 August 2026 for submission to the IESG.

That is two days from the date on this post. So we audited them.

Two documents, one deadline

The working group's document page lists draft-ietf-aipref-vocab, now at revision 07, and draft-ietf-aipref-attach, at revision 05. Both are Standards Track. Both were refreshed on 18 August 2026 and carry the in-document date 19 August 2026. Both now expire 20 February 2027.

The revision history is the first thing worth reading. Both documents entered In WG Last Call on 4 September 2025. Both were moved back to WG Document on 3 November 2025. Neither has re-entered last call. On the datatracker their IESG state today is I-D Exists, which is the state meaning the IESG has not started processing them.

The attachment draft then went quiet. Revision 04 was posted on 28 October 2025; the document expired on 1 May 2026; revision 05 arrived on 18 August 2026. Nine and a half months between revisions, three and a half of them expired, for the document that defines the HTTP field name and the robots.txt directive everyone is supposed to emit.

The vocabulary draft is the more remarkable of the two. Since revision 06 it carries a Note to Readers, ahead of the Status of This Memo, which we quote in full:

As detailed below, this is a working document. Its contents DO NOT REFLECT CONSENSUS of the Working Group either in whole or part. Presense or absense of any particular text does not indicate consensus, and this document is published solely as a basis of further discussion.

Section 4, the section that actually defines the categories, repeats it locally: "NOTE: This section does not yet have consensus." We checked every revision from 00 through 07. The note appears first in revision 06, posted 27 April 2026, and survives into revision 07. It does not appear in the attachment draft at all.

The split is informative. Where to put the preference is settled. What you are allowed to say is not. And the thing an autonomous agent needs is the second one.

The vocabulary got smaller

Fourteen months of revisions moved in one direction. We pulled section 4 out of every published revision:

// draft-ietf-aipref-vocab, section 4, by revision

-01  Text and Data Mining · AI Training · Generative AI Training
     · Search · AI Inference
-02  Automated Processing · AI Training · Generative AI Training
     · AI Use · Search
-03  Automated Processing · AI Training · Generative AI Training · Search
-04  Automated Processing · Foundation Model Production
     · AI Output · Search
-05  Foundation Model Production · Search
-07  AI Model Training (train-ai) · Search (search)

Revision 01, from 18 June 2025, defined AI Inference as "the act of using one or more assets as input to a trained AI/ML model as part of the operation of that model (as opposed to the training of the model)." That sentence is a precise description of what happens when an agent fetches a page and passes it to a model through a gateway. It was dropped in revision 03. Its successors under other names — AI Use in 02, AI Output in 04 — were dropped in turn.

What survives in revision 07 is two categories. AI Model Training is "using an asset to modify the learned parameters of an AI model that is used to generate synthetic content in one or more modalities." Search is use in an application "where the primary purpose of the application is to select assets and direct users to the location of those assets", conditional on linking back and on excerpts serving relevance evaluation. And then, explicitly: "This category does not include the use of assets to generate summaries."

So a summarizing agent is not doing Search. It is also not doing training. Under the vocabulary the IETF is two days from submitting, an autonomous agent that fetches a page to answer a question falls into no defined category at all.

The word "agent" does not appear anywhere in any revision of the vocabulary draft.

Revision 07 removed the ladder

Until revision 06 there was a mechanism that partially covered the gap. Categories nested, and preferences flowed downhill. Revision 06 section 5 resolved a preference in three steps: explicit statement wins; otherwise, if the category is a proper subset of another, recurse into the parent; otherwise unknown. Extensions were constrained to match: "Any future extensions to this vocabulary MUST NOT introduce additional categories that include existing categories defined in the vocabulary." Subsets only, never supersets.

Revision 07 deletes both halves. The resolution algorithm is now two sentences with no recursion: explicit statement wins, "Otherwise, the preference for that category is unknown." The extension rule became "The definition of the extension MUST define how any potential overlap between usage categories is resolved."

The consequence is concrete. Under revision 06, a publisher who wrote a broad disallow would have covered narrower categories defined later. Under revision 07, every future category starts at unknown, no matter what the publisher already said. And the document declines to fill that hole: "One approach for dealing with an unknown outcome is to assign a default value. This document takes no position on what default might be assigned."

Revision 07 also rewrote the scope section into a list of things the specification does not do. It does not "ensure that preferences are followed", does not "address if, how, or when preferences should be followed or not-followed", and does not "address technical, legal, contractual, or other mechanisms that might create a stronger requirement to follow or not follow preferences."

The attachment draft made a smaller move in the same direction. Revision 04 said "Servers MUST retain any preferences associated with a request" when that content is later served. Revision 05 says "Servers can use any preferences associated with a request." A MUST became a can.

What the web actually says

Specifications are cheap to read and expensive to verify. So on 29 August 2026 we crawled the top 10,000 domains of the Majestic Million, requesting /robots.txt over HTTPS with redirects followed, and counting only responses that returned status 200 with a non-HTML content type. 6,760 domains qualified.

The three Content-Usage deployments are worth naming, because there are only three. launchpad.net and semafor.com both emit Content-Usage: ai=n. That key does not exist in the current vocabulary. It is the example from draft-ietf-aipref-attach-00, dated 27 May 2025, superseded by train-ai=n in revision 02 fourteen months ago. Both sites cite the same source in a robots.txt comment: robotstxt.com/ai, a template generator that describes Content-Usage as a directive "Google has proposed" and ships the stale example.

The third is follow.it, which emits Content-Usage: bots=y, train-ai=n inside a block marked # START nuxt-robots. It is generated by the @nuxtjs/robots module, whose documentation states that the directive "follows the IETF AI Preferences specification" and offers four categories: bots, train-ai, ai-output, search. Two of those are not in the specification. ai-output existed only in revision 04, between October and December 2025. bots never existed in any revision.

In the top 1,000 subset, 741 domains had a readable robots.txt. One served a Content-Signal HTTP response header. None served a Content-Usage header. We also checked the IANA HTTP Field Name registry on the same day: 260 entries, and neither Content-Usage nor Content-Signal is among them. The attachment draft's entire IANA considerations section is that one registration, still pending.

Running the specification against the wild

Counting directives is not the same as reading them. Revision 07 section 6 defines the serialization as an RFC 9651 structured-field dictionary: keys are the category labels, values are the single-byte tokens y and n, and section 6.4 says unknown labels MUST be ignored. So we implemented sections 5 and 6 in about thirty lines and ran every directive we collected through it.

from http_sfv import Dictionary, Token

CATEGORIES = ("train-ai", "search")   # vocab-07, section 4

def resolve(field_value):
    d = Dictionary(); d.parse(field_value.encode("ascii"))
    out = {}
    for c in CATEGORIES:
        v = getattr(d.get(c), "value", None)
        # absent key, non-token value, or any token other than y/n -> unknown
        out[c] = {"y": "allow", "n": "disallow"}.get(
            str(v) if isinstance(v, Token) else None, "unknown")
    ignored = [k for k in d.keys() if k not in CATEGORIES]
    return out, ignored

Against the 419 Content-Signal lines we collected from 396 domains, the result is unambiguous: zero produce a defined preference for either category. Two independent mismatches cause it. The key is reversed — Cloudflare writes ai-train, the vocabulary defines train-ai, so 413 of the lines contribute a key that must be ignored. And the value alphabet differs — across all 419 lines we counted 598 instances of yes, 376 of no, 268 of reference, and zero of y or n. Even search, the one label both vocabularies spell the same way, resolves to unknown, because yes is a perfectly valid structured-field token that simply is not y.

The Content-Usage lines fare marginally better. Two resolve to unknown for both categories, because ai is an ignored key. One resolves: follow.it yields train-ai=disallow, and search=unknown.

One domain in the top 10,000, from a framework default. That is the current legible surface of the IETF AI-preferences vocabulary.

The vendor spec has the category agents need

The asymmetry is not just adoption. It is expressiveness. Cloudflare's managed robots.txt documentation, last updated 3 August 2026, defines three signals. search and ai-train map roughly onto the IETF pair. The third does not map onto anything:

ai-input: inputting content into one or more AI models (e.g., retrieval augmented generation, grounding, or other real-time taking of content for generative AI search answers).

That is the deleted AI Inference category, alive in production. In the top 1,000, 16 of the 23 Content-Signal domains express a preference about ai-input. Across the full 10,000, the key appears 135 times. Cloudflare is also testing a fourth field, content-use, with three values — immediate (interact but store nothing), reference (index, excerpt, link back), full (summarize and reproduce) — and ships use=reference in the managed default. We counted 268 instances of it in the wild.

The framing differs as much as the vocabulary. Cloudflare's managed block opens with "As a condition of accessing this website, you agree to abide by the following content signals" and closes by declaring restrictions to be express reservations of rights under Article 4 of EU Directive 2019/790. It is written as a contract. The IETF vocabulary explicitly declines to address contractual mechanisms, and the AIPREF charter puts "technical enforcement of preferences", "application layer protocols for authenticating or authorizing clients and/or crawlers", "establishment of registries of preferences relating to content", and "auditing and other transparency measures for AI training" out of scope.

Cloudflare's documentation does not mention the IETF, AIPREF, or Content-Usage anywhere. We checked the rendered page: zero occurrences of each.

The registry exclusion deserves its own sentence. Because registries are out of charter scope, revision 07 states "This document has no IANA actions", and extending the vocabulary requires "a standards-track RFC that updates this document." Adding a category for agent retrieval is not a registration. It is an RFC.

Nobody in the working group is talking about money

Meanwhile, the things the charter excludes are being written anyway, as individual submissions.

draft-reilly-aipref-compliance-00, dated 2 August 2026, defines an AI Usage Compliance Record: a structure binding a retrieved asset, the preference in force at retrieval time, and the usage category the processing entity assigned, with an aggregation scheme so one signature covers very large numbers of records. That is the audit trail the charter excludes.

draft-wallace-aipref-grant-binding-02, dated 18 August 2026, is closer to our problem. It observes that a preference "expresses a reservation. It does not, by itself, provide a verifiable, revocable record of a specific grant that lifts a preference for a specific party." It proposes an offline-verifiable, revocable credential that names a party, an asset, and a usage category. It also states plainly that it "is not an enforcement or access-control mechanism."

A grant that lifts a reservation for a named party is, economically, a purchase. So we grepped for the other half. Across the ten documents on the AIPREF page — the two working group drafts and the eight individual submissions — x402 appears zero times, 402 as a status code appears zero times, micropayment appears zero times, and the word payment appears exactly once: in draft-reilly-aipref-compliance-00, in the list of things it does not cover.

The reverse grep is just as clean. The x402 monorepo at HEAD e398a9e, 28 August 2026, has zero occurrences of aipref, content-usage, content-signal, train-ai, or even robots.txt. The MCP specification repository at HEAD ca4ab30 and the A2A repository at HEAD f63dbb4 have zero occurrences of aipref.

Two layers that need each other, with no reference in either direction.

What it means for LLM4Agents

LLM4Agents is the gateway an agent calls when it wants a model, and the rail it pays over. When an agent fetches a page and pushes the text into a completion, our gateway is the processing entity in AIPREF's terms. That places us on the receiving side of every one of these findings.

The first is that the signal is nearly unreadable today. A conforming vocab-07 implementation would extract a preference from one domain out of the top 10,000. Anyone shipping AIPREF support alone and calling the retrieval path compliant is shipping a no-op. The real signal is elsewhere. We parsed the rule groups of the 737 top-1,000 domains with a readable robots.txt: 250 name a training crawler and 199 serve it a full Disallow: /, while 195 name an agent-class fetcher and 138 fully block one. 62 of them block trainers and leave every agent fetcher unblocked. Publishers are drawing the train-versus-read line the vocabulary cannot express, and they are drawing it with bot names.

The second is that the category we operate in does not exist in the standard. Our traffic is inference-time retrieval, and the IETF deleted that category in September 2025 and removed, in August 2026, the subsumption rule that would have let a broader preference cover it by implication. Under revision 07 the honest answer for an agent-retrieval fetch is unknown, with the document declining to say what unknown defaults to. That is a policy decision pushed onto the gateway operator, and we should make it deliberately rather than by omission.

The third is where the opportunity sits. Grant binding describes a credential that lifts a reservation for a named party, and refuses to say how that party comes to deserve it. x402 answers exactly that question for a walk-up client with no prior relationship: quote a price in the 402, settle in stablecoins, serve the content. A publisher who writes ai-input=no today is saying no because the only alternative is unpriced consumption. A 402 turns that into a price. The gap between the two documents is a market, and neither side has written the bridge.

Staying on the frontier

Concrete, ordered by what we would do first.

Read all three vocabularies, resolve deterministically, log the disagreement. Our fetch path should parse Content-Usage, Content-Signal, and RSL's License:, and record which of them spoke for each retrieval. The crawl says the coverage ratio is 132 to 1 in the vendor's favour, so an implementation that reads only the standards-track field is blind by construction. Vocabulary revision 07 section 6.6 permits alternative formats provided the mapping is defined; nobody has defined the Content-Signal mapping, so we should publish ours — key rename, value alphabet, and an explicit statement that ai-input has no IETF equivalent.

Pick an explicit default for unknown and put it in the API. The specification refuses to. Every retrieval through our gateway should carry a resolved triple — allow, disallow, unknown — for training, search, and inference-time input, and unknown must resolve to a documented posture the caller can see and override. Silence is not consent, and it is not refusal either; it is a field the agent operator has to be able to reason about.

Emit the compliance record before anyone requires it. The reilly draft's structure — asset, preference in force at retrieval, category assigned — is close to what our billing path already knows, because we already record what was fetched and what was charged. Binding the preference we read to the payment we settled is a small addition to an existing receipt, and it is the artefact that turns "we honour preferences" into something a publisher can verify. It composes with the telemetry attributes we already emit.

Prototype the grant-for-payment path against a real publisher. Take one domain that publishes ai-train=no, offer a priced exception over x402, and settle it. The grant-binding draft supplies the credential shape; x402 supplies the rail; the missing piece is a resource server that treats a paid settlement as the event that issues the credential. That is a weekend prototype and a genuinely new data point, because no draft in either ecosystem currently references the other.

Track the 31 August milestone and the last-call state, not the revision number. The signal to watch is not a new draft. It is whether the vocabulary re-enters WG Last Call and whether the Note to Readers disappears. Until that note is gone, section 4 is not settled, and any category we build product assumptions on can still change.