Skip to main content

Solutions

Enforcing robots.txt

robots.txt asks politely and the operators who ignore it are the ones you wrote it for. How to enforce the same policy at the origin and keep the two in step.

Start free Browse the catalogue

Free plan, no card. 100,000 events a month with every feature on.

botscope.io · last 7d

This site, classified by this product. Nothing here is a mock-up.

live
requests
4,245
human
80%
AI agents
0%
stopped
0%
verdict stream
  1. signals
    • threat_feed: datacenter — fail (listed on a subscribed feed)
    • anomaly_score: 0.25 — pass (combined disagreement across the collected signals)
  2. signals
    • threat_feed: datacenter — fail (listed on a subscribed feed)
    • anomaly_score: 0.25 — pass (combined disagreement across the collected signals)
  3. signals
    • anomaly_score: 0.40 — pass (combined disagreement across the collected signals)

botscope.io's own traffic only — never a customer's. No addresses, no query strings, and a path is shown only when it is already a public page.

In short

  • robots.txt is a request. It has no enforcement of its own and never has.
  • The operators most likely to ignore it are the ones the rule was written for.
  • Botscope generates robots.txt from the same per-agent policy it enforces, so the published rule and the applied one cannot drift.
  • A published rule that is not enforced is still worth having: it is what makes ignoring it deliberate.

Updated 16 September 2026

01 · two files that disagree

Two files that disagree

The common failure is not a missing robots.txt. It is a robots.txt written eighteen months ago that names agents that have been renamed, and a set of firewall rules written since that say something else.

Neither is wrong on its own. Together they mean nobody can answer what the site's policy actually is.

02 · keeping one policy

Keeping one policy

  1. 1 Set the policy per agent once, in one place.
  2. 2 Generate robots.txt from that policy rather than editing it by hand.
  3. 3 Enforce the same policy at the origin for agents that ignore the file.
  4. 4 When an operator renames an agent, the catalogue entry updates and both outputs follow.

03 · agents with a declared robots.txt token

Agents with a declared robots.txt token

Bot Operator Default policy Verifiable
AI Search Cloudflare Observe By user agent
AIWebIndex-Agent Lyrenth Observe By user agent
Amazon Kendra Amazon Observe By user agent
Amazonbot Amazon Observe By user agent
Amzn-SearchBot Amazon Observe By user agent
Anchor Browser Anchor Observe Signed requests
Anomura Direqt Observe By user agent
Applebot Apple Observe By user agent
Atlassian (Rovo AI) Atlassian Allow By user agent
atlassian-bot Atlassian Observe By user agent

Read live from the Botscope crawler catalogue. Default policy is the catalogue default; every agent can be overridden per site.

04 · where enforcement does not reach

Where enforcement does not reach

An agent that does not identify itself cannot be matched by name, in the file or at the origin. It is caught, if at all, by behaviour — which is a different mechanism with different trade-offs, including false positives you have to be willing to look at.

Questions

Is robots.txt legally binding?

It is not a contract, and this is not legal advice. Its practical value is that it makes a boundary explicit, which matters for the conversation that follows when someone crosses it.

Should I keep robots.txt if I enforce at the origin?

Yes. Well-behaved operators read it and stop before they cost you anything, and it documents your intent.

What about Crawl-delay?

Support is inconsistent between operators. A rate policy at the origin applies regardless of whether the directive was read.

Does the generated file overwrite mine?

Only if you choose to serve the generated one. You can also copy it and keep serving your own file.

See which of these reach your site

Free plan, no card: 100,000 events a month with every feature on. Nothing is blocked until you turn enforcement on.

Start free Browse the catalogue

Related