Updated 16 September 2026
01 · what the file is
What the file is
llms.txt is a plain markdown file served at the root of a domain — /llms.txt — that lists the pages a site considers authoritative, with a short line on what each is for. The idea is straightforward: a language model answering a question about you has to pick something to read, and left to itself it will pick whatever ranked, which is often a changelog, a pricing page from two years ago, or somebody else writing about you.
The file is a proposal. It has no governing body, no compliance requirement and no mechanism that makes an agent fetch it. Treating it as a standard is the most common mistake made about it.
What it does have is a low cost. It is a text file. Writing one takes an afternoon and it cannot break anything.
02 · how to write one
How to write one
Keep it short and keep it true. A file that lists forty pages tells a model nothing about which matter.
Start with an H1 that is the name of the project, then one blockquote line saying what it is. Then a handful of H2 sections — documentation, product, policies — with markdown links and a clause each on why that page exists. Put the pages you would hand a new employee in their first week; leave out anything you would not want quoted back to you.
Serve it as text/markdown or text/plain, at the root, at /llms.txt, without a redirect. Keep it in the same repository as the site so it is updated by whoever updates the pages it points at, rather than by whoever remembers it exists.
03 · against robots.txt
Against robots.txt
robots.txt is older, widely honoured by search crawlers, and answers a permission question: may this path be fetched. It is a list of prohibitions and it is voluntary too, but the convention around it is decades old and the cost of ignoring it is reputational.
llms.txt answers a curation question: of everything you may fetch, what should you read. It grants nothing and forbids nothing.
The practical consequence is that they do not substitute. A site that wants an AI crawler to stay out of a directory needs robots.txt, and possibly enforcement behind it. A site that wants to be described accurately when it is read needs llms.txt. Most sites that care about one care about the other.
04 · the agents that would be reading it
The agents that would be reading it
These are AI clients Botscope names from our catalogue, with the verification method each operator publishes. Whether any given one fetched your llms.txt is a question about your logs, not about this table.
| Bot | Operator | Default policy | Verifiable |
|---|---|---|---|
| AI Search | Cloudflare | Observe | By user agent |
| AIWebIndex-Agent | Lyrenth | Observe | By user agent |
| Amazon Kendra | Amazon | Observe | By user agent |
| Amazonbot | Amazon | Observe | By user agent |
| Amzn-SearchBot | Amazon | Observe | By user agent |
| Anchor Browser | Anchor | Observe | Signed requests |
| Anomura | Direqt | Observe | By user agent |
| Applebot | Apple | Observe | By user agent |
| Atlassian (Rovo AI) | Atlassian | Allow | By user agent |
| atlassian-bot | Atlassian | Observe | By user agent |
| AwarioSmartBot | Awario | Observe | By user agent |
| AzureAI-SearchBot | Microsoft | Observe | By user agent |
Read live from the Botscope crawler catalogue. Default policy is the catalogue default; every agent can be overridden per site.
05 · what publishing one does not get you
What publishing one does not get you
It does not stop anybody fetching anything. It does not remove your pages from a training set. It does not put you in an answer, and a model that has never fetched your domain will not find you because the file exists.
There is also no way to tell from the file whether it worked. A fetch of /llms.txt appears in your request log like any other request, with a user agent that may or may not be honest, from an address that may or may not belong to who it claims. Answering "did anyone read it" means classifying that request, which is the part the file cannot do for you.
Questions
Is llms.txt an official standard?
No. It is a proposed convention with growing adoption and no governing body. Nothing obliges an agent to request it, and no agent is in breach for ignoring it.
Does it replace robots.txt?
No. robots.txt is about permission to fetch; llms.txt is about what is worth reading. Publishing one does not affect the other, and a site that wants crawlers kept out of a path still needs robots.txt.
How do I know whether an AI agent read mine?
By looking at requests for /llms.txt in your own logs and establishing who actually made them. A user agent string is self-reported; the operators worth trusting publish address ranges or reverse-DNS masks that let you check. The catalogue lists which ones do.
Will publishing one get me into ChatGPT answers?
Not by itself. It improves the odds that a model reading your site reads the pages you would have chosen. It does nothing about whether a model reads your site at all.
Where should it live?
At /llms.txt on the root domain, served as markdown or plain text, without a redirect, and kept in the repository alongside the pages it points at.
See which of these reach your site
Free plan, no card: 100,000 events a month with every feature on. Nothing is blocked until you turn enforcement on.