Skip to main content

Solutions

Blocking AI scrapers

Not every AI crawler wants the same thing. Separate training collection from assistant fetches and search indexing, then set a policy for each one deliberately.

Start free Browse the catalogue

Free plan, no card. 100,000 events a month with every feature on.

botscope.io · last 7d

This site, classified by this product. Nothing here is a mock-up.

live
requests
4,245
human
80%
AI agents
0%
stopped
0%
verdict stream
  1. signals
    • threat_feed: datacenter — fail (listed on a subscribed feed)
    • anomaly_score: 0.25 — pass (combined disagreement across the collected signals)
  2. signals
    • threat_feed: datacenter — fail (listed on a subscribed feed)
    • anomaly_score: 0.25 — pass (combined disagreement across the collected signals)
  3. signals
    • anomaly_score: 0.40 — pass (combined disagreement across the collected signals)

botscope.io's own traffic only — never a customer's. No addresses, no query strings, and a path is shown only when it is already a public page.

In short

  • Blanket-blocking AI agents also removes the ones that send readers back to you.
  • Botscope separates training collection, live assistant fetches and search indexing, so each gets its own policy.
  • A robots.txt rule is a request. Enforcement at the origin is what makes it a boundary.
  • Observe first: you can see what a policy would have done before anything is refused.

Updated 16 September 2026

01 · three different things wearing one label

Three different things wearing one label

"AI bot" covers at least three unrelated behaviours. A training collector reads your pages into a dataset and returns nothing. An assistant fetches a page because a person asked about it right now, and that person may click through. A search-adjacent crawler indexes you the way search always has.

Blocking all three is a decision, and a defensible one. It is only a bad decision when it is made by accident because the categories were never separated.

02 · ai agents, by what they actually do

AI agents, by what they actually do

Bot Operator Default policy Verifiable
AI Search Cloudflare Observe By user agent
AIWebIndex-Agent Lyrenth Observe By user agent
Amazon Kendra Amazon Observe By user agent
Amazonbot Amazon Observe By user agent
Amzn-SearchBot Amazon Observe By user agent
Anchor Browser Anchor Observe Signed requests
Anomura Direqt Observe By user agent
Applebot Apple Observe By user agent
Atlassian (Rovo AI) Atlassian Allow By user agent
atlassian-bot Atlassian Observe By user agent
AwarioSmartBot Awario Observe By user agent
AzureAI-SearchBot Microsoft Observe By user agent

Read live from the Botscope crawler catalogue. Default policy is the catalogue default; every agent can be overridden per site.

03 · setting a policy you can defend

Setting a policy you can defend

  1. 1 Start in observe. Every agent is recorded, nothing is refused.
  2. 2 Read a week and see who actually arrives. Most sites are surprised in both directions.
  3. 3 Set a policy per category: allow, observe, challenge or block.
  4. 4 Publish the matching robots.txt from the same policy, so the declared rule and the enforced one cannot drift.
  5. 5 Turn enforcement on when the preview matches your intent.

04 · what blocking cannot do

What blocking cannot do

It cannot remove content already collected. A block applies from the moment it is on.

It cannot stop a determined scraper that rotates residential addresses and drives a real browser. It raises the cost, which is the honest description of what any of this does.

Questions

Will blocking AI crawlers hurt my search ranking?

Search crawlers and AI training collectors are separate agents with separate policies here, so blocking the latter does not touch the former. Some operators use one agent for both purposes — those are labelled in the catalogue, because for them it is genuinely a trade-off.

Is robots.txt enough on its own?

It is a request that well-behaved operators honour. Enforcement at the origin is what applies to the ones that do not.

Can I allow an assistant but block training?

Yes, where the operator uses distinct agents for the two, which most now do. The catalogue records which agent does which.

How do I know a request is really from the operator it claims?

Reverse DNS, published address ranges, or a request signature where the operator supports one. The catalogue records which method applies to each agent.

See which of these reach your site

Free plan, no card: 100,000 events a month with every feature on. Nothing is blocked until you turn enforcement on.

Start free Browse the catalogue

Related