Web Bot Auth
An emerging web standard that lets AI agents identify themselves cryptographically — HTTP requests signed with published keys, so "which agent is this?" stops being guesswork.
01What it means
Web Bot Auth is an emerging web standard (an IETF draft) for agent identification: automated agents sign their HTTP requests with cryptographic keys, and sites can verify which agent — operated by whom — is asking. It moves bot traffic from user-agent-string cosplay to verifiable identity: a request that says it comes from a particular company's agent can be checked against that company's published keys.
For sites, verification is what unlocks real policy. You cannot enforce "verified search agents welcome, unknown scrapers throttled" until you can tell them apart — and user-agent strings are trivially forged while IP allowlists rot as infrastructure churns. Keys travel with the request. The key directory lives at a well-known location; this site publishes one so visiting agents can be checked.
It is the trust layer the rest of the agent-facing web leans on: llms.txt and agent cards publish offers and information; Web Bot Auth makes the identity behind a request checkable. None of the protocols here require it, and all of them get better with it.
02Why it matters for buyers
- —Policy needs identity: "agents welcome, abuse not" is unenforceable without a way to tell one agent from another — signed requests are that way.
- —It is early but cheap: publishing a key directory is a small, visible commitment to doing agent traffic properly, and it costs one well-known file.
- —It is on the checklist: our agent-readiness tool checks for a Web Bot Auth key directory on any public site.
03Common questions
How does Web Bot Auth differ from robots.txt?
robots.txt is a request — please do not crawl this. Web Bot Auth is a credential — prove who you are. One is policy you can only publish; the other is policy you can actually enforce.
Does it replace IP allowlists?
It complements them. Keys travel with the request and survive the infrastructure churn that makes IP lists stale; IPs remain useful as a coarse filter underneath.
Where do the keys live?
In a well-known key directory on the site — /.well-known/http-message-signatures-directory in the current draft — which agents fetch and verify signed requests against. This site publishes one.