Skip to main content

By platform

Botscope for SEO teams

See which crawlers read which pages, how often, and what came back. Per-page crawl and referral figures, without a tag that automated clients never execute.

Start free Browse the catalogue

Free plan, no card. 100,000 events a month with every feature on.

botscope.io · last 7d

This site, classified by this product. Nothing here is a mock-up.

live
requests
4,245
human
80%
AI agents
0%
stopped
0%
verdict stream
  1. signals
    • threat_feed: datacenter — fail (listed on a subscribed feed)
    • anomaly_score: 0.25 — pass (combined disagreement across the collected signals)
  2. signals
    • threat_feed: datacenter — fail (listed on a subscribed feed)
    • anomaly_score: 0.25 — pass (combined disagreement across the collected signals)
  3. signals
    • anomaly_score: 0.40 — pass (combined disagreement across the collected signals)

botscope.io's own traffic only — never a customer's. No addresses, no query strings, and a path is shown only when it is already a public page.

In short

  • Crawl data per page, from your own logs rather than from a sample.
  • AI assistants are separated from search crawlers, because the two answer different questions.
  • Pages that are read heavily and send nobody back are visible as such.
  • No claim about rankings: this reports what reached your server.

Updated 16 September 2026

01 · the gap between the console and the server

The gap between the console and the server

Search consoles report a subset of one operator's crawling, on their schedule. They say nothing about the AI agents now reading the same pages.

Your origin sees all of it, unsampled and immediately. The work is turning that into something per-page rather than a log.

02 · crawlers separated by what they are for

Crawlers separated by what they are for

Bot Operator Default policy Verifiable
360Spider Qihoo 360 Allow By user agent
Alexa Archive Alexandria.org Allow By user agent
alexa site audit Amazon Allow By user agent
AlexandriaOrgBot Alexandria.org Allow By user agent
Algolia Algolia Allow By user agent
All Africa Crawler AllAfrica Global Media Allow By user agent
Atom Feed Robot RSSMicro Allow By user agent
Baidu ADS Server Proxy Baidu Allow By user agent
Baiduspider Baidu Allow By user agent
Baiduspider-render Baidu Allow By user agent
bingbot Microsoft Allow By user agent
BingBot Microsoft Allow By user agent

Read live from the Botscope crawler catalogue. Default policy is the catalogue default; every agent can be overridden per site.

03 · what to look at first

What to look at first

  1. 1 Pages crawled often that receive no human visits: read, and not sending anybody.
  2. 2 Pages crawled by assistants but not by search crawlers, and the reverse.
  3. 3 New pages, and how long until each operator picks them up.
  4. 4 Impostor traffic claiming to be a search crawler and failing verification.

04 · what this is not

What this is not

It is not a rank tracker and does not model ranking factors. It reports requests that reached your origin and visits that followed. Anything beyond that would be inference presented as measurement.

Questions

Does this replace Search Console?

No. Search Console reports things only the operator knows, such as impressions. This reports what actually hit your server, across every operator.

Can I export it?

Yes, through the API, which is available on the free plan.

Is the crawl data sampled?

No. It is your own request data.

How far back does it go?

As far as your plan’s retention. Aggregates are kept longer than individual requests.

See which of these reach your site

Free plan, no card: 100,000 events a month with every feature on. Nothing is blocked until you turn enforcement on.

Start free Browse the catalogue

Related