What robots.txt actually is
robots.txt is a plain-text file at the root of your domain (yoursite.com/robots.txt) that lists which paths a crawler is asked not to fetch. It's part of the Robots Exclusion Protocol, an informal convention dating back to 1994 that Google formally proposed as an internet standard in 2019 — it became RFC 9309 in 2022.
The key word in that history is convention. There is nothing in HTTP that requires a client to fetch robots.txt before requesting a page, and nothing that stops it from fetching a disallowed path anyway. The file works purely because the crawlers reading it choose to respect what it says.
Who still follows the rules
The crawlers with the most to lose from being seen as bad actors are also the most reliable about honoring robots.txt:
- Googlebot and Bingbot — both search engines have a direct commercial incentive to be trusted crawlers, and both publish detailed, accurate documentation of how they parse the file.
- Established SEO and monitoring tools (site auditors, uptime checkers) that depend on staying in webmasters' good graces.
- Most reputable AI crawlers that also publish their own documentation and IP ranges, when the operator has a long-term reputation to protect.
Who doesn't — and why
The past two years have seen a sharp rise in a category that doesn't fit the old assumptions: LLM training and retrieval crawlers. Some, like GPTBot and ClaudeBot, publish documentation and generally respect Disallow rules. Others have been widely reported to ignore blocks, rotate IPs and user-agent strings to route around them, or simply not check the file at all before crawling.
There's also a structural reason this is getting worse, not better: the incentive to scrape has grown much faster than the reputational cost of ignoring the rules. Training data and real-time retrieval for AI answers are valuable enough that a growing set of operators — from well-funded labs to anonymous scraping-for-hire services — treat robots.txt as advisory at best.
And underneath all of that sits a much older problem: anyone can claim to be anyone. A scraper that wants your content regardless of what your file says can simply not send a matching User-Agent string, or send one that impersonates a bot you've chosen to allow. See our companion piece on verifying Googlebot for exactly how trivial that impersonation is.
Why it was never meant to be enforced
It's worth being precise about what robots.txt was designed to solve: in 1994, the problem was well-meaning crawlers accidentally hammering servers or indexing pages their operators didn't want indexed. It was a courtesy signal between cooperating parties, not a security boundary. It has:
- No authentication — anyone can read it, and nothing verifies who's asking.
- No enforcement mechanism — the server serves the file and then serves the disallowed page anyway, if asked.
- No way to verify the requester's claimed identity — a Disallow rule aimed at "BadBot" does nothing to a crawler that simply doesn't call itself that.
Treating it as a security control is a category error — it's closer to a "Please don't pick the flowers" sign than a fence.
What actually works instead
If you actually need to control who accesses what — not just politely ask — the enforcement has to move to where you control the outcome: your own server or edge, independent of what the requester claims about itself.
- Verify identity cryptographically or via DNS, not by trusting the User-Agent header. Reverse DNS + forward confirmation, published IP ranges, and the newer Web Bot Auth signature standard all verify who's actually asking — see our full comparison of the three methods.
- Rate-limit and fingerprint independent of self-reported identity. A crawler that lies about who it is still has to make requests from somewhere, at some rate, with some client behavior — anomaly detection on those signals catches what an honor-system file cannot.
- Skip the interruption when you can. The same layered-signal approach that verifies identity can also score ordinary visitors invisibly — see our piece on why proactive protection beats CAPTCHA for how that applies beyond crawlers.
- Apply a real policy per verified identity — allow, observe, challenge, or block — at the point of request, so the decision is made by your server, not by whether the visiting bot decided to cooperate.
This is the gap Botscope fills: verified crawlers get an automatic pass, unknown automation gets watched and scored, and anything actively abusive gets challenged or blocked — all enforced at your edge, regardless of whether the requester ever looked at your robots.txt at all.
What about llms.txt?
A newer convention, llms.txt, has emerged as a way for sites to tell AI systems what content they may use, formatted specifically for language models rather than search-engine crawlers. It's worth publishing alongside robots.txt as another cooperative signal — but it inherits exactly the same limitation: it's read and honored voluntarily, by crawlers that choose to look for it, with no technical mechanism forcing compliance. Treat it the same way — a courtesy, not a control.
FAQ
Is ignoring robots.txt illegal?
No. It's a voluntary convention, not a legal or technical access control. Ignoring it isn't illegal by itself, though it may become relevant to unrelated disputes over terms-of-service violations or unauthorized access, depending on your jurisdiction and what else the crawler did.
Should I still publish a robots.txt file?
Yes. Search engines and most well-behaved crawlers still read and follow it, so it remains genuinely useful for SEO and for signaling crawl preferences to cooperative bots. Just don't rely on it as your only, or even your main, defense against unwanted scraping.
Does blocking a bot in robots.txt actually stop it from accessing my site?
Only if that bot chooses to honor the file. Nothing in HTTP enforces a Disallow rule — it's fetched and interpreted voluntarily by the crawler's own code, so any bot that ignores it faces zero technical barrier.