Skip to main content

By platform

Botscope for publishers

Separate the AI agents that send readers back from the ones that only collect, then set a policy per operator and enforce the same rule you publish.

Start free Browse the catalogue

Free plan, no card. 100,000 events a month with every feature on.

botscope.io · last 7d

This site, classified by this product. Nothing here is a mock-up.

live
requests
4,245
human
80%
AI agents
0%
stopped
0%
verdict stream
  1. signals
    • threat_feed: datacenter — fail (listed on a subscribed feed)
    • anomaly_score: 0.25 — pass (combined disagreement across the collected signals)
  2. signals
    • threat_feed: datacenter — fail (listed on a subscribed feed)
    • anomaly_score: 0.25 — pass (combined disagreement across the collected signals)
  3. signals
    • anomaly_score: 0.40 — pass (combined disagreement across the collected signals)

botscope.io's own traffic only — never a customer's. No addresses, no query strings, and a path is shown only when it is already a public page.

In short

  • Training collection and reader-sending assistants are different agents and deserve different answers.
  • A per-operator policy is the practical basis for a licensing conversation.
  • Crawl volume per page, next to what came back, shows which content is carrying the trade.
  • Enforcement is off until you turn it on, and observe shows what it would have done.

Updated 16 September 2026

01 · the trade is not the same for every operator

The trade is not the same for every operator

Some AI operators send readers back through citations. Some collect and return nothing. Some do both with separate agents, and a few with the same one.

A single blanket decision treats these as one thing. The per-operator picture is what makes it possible to allow the ones that send traffic and refuse the ones that do not — and to have that conversation with a number in hand.

02 · ai operators reading editorial content

AI operators reading editorial content

Bot Operator Default policy Verifiable
AI Search Cloudflare Observe By user agent
AIWebIndex-Agent Lyrenth Observe By user agent
Amazon Kendra Amazon Observe By user agent
Amazonbot Amazon Observe By user agent
Amzn-SearchBot Amazon Observe By user agent
Anchor Browser Anchor Observe Signed requests
Anomura Direqt Observe By user agent
Applebot Apple Observe By user agent
Atlassian (Rovo AI) Atlassian Allow By user agent
atlassian-bot Atlassian Observe By user agent
AwarioSmartBot Awario Observe By user agent
AzureAI-SearchBot Microsoft Observe By user agent

Read live from the Botscope crawler catalogue. Default policy is the catalogue default; every agent can be overridden per site.

03 · building the position

Building the position

  1. 1 Observe for a period long enough to cover your publishing rhythm.
  2. 2 Read crawl volume per operator, and referred visits per operator beside it.
  3. 3 Set a policy per operator rather than per category.
  4. 4 Publish the matching robots.txt, so the declared boundary and the enforced one are the same one.

04 · what this does not settle

What this does not settle

Nothing here is a legal position and none of it is legal advice. It is evidence about what was read and what came back, which is the input to that conversation rather than the outcome of it.

Questions

Will blocking AI crawlers cost me search traffic?

Not where the operator separates its search crawler from its training collector, which most now do. Where one agent serves both, the catalogue says so, because there the trade-off is real.

Can I allow one operator and refuse another?

Yes. Policy is per agent, and agents are grouped by operator.

Does this help with a licensing negotiation?

It gives you what was crawled and what came back, per operator. What that is worth is your negotiation.

What about content already collected?

A policy applies from the moment it is on. Nothing retroactive is possible here or anywhere else.

See which of these reach your site

Free plan, no card: 100,000 events a month with every feature on. Nothing is blocked until you turn enforcement on.

Start free Browse the catalogue

Related