Skip to main content

Solutions

Content scraping

Scraping cannot be made impossible, only expensive. What actually raises the cost, what only moves the problem, and how to tell which is happening on your site.

Start free Browse the catalogue

Free plan, no card. 100,000 events a month with every feature on.

botscope.io · last 7d

This site, classified by this product. Nothing here is a mock-up.

live
requests
4,245
human
80%
AI agents
0%
stopped
0%
verdict stream
  1. signals
    • threat_feed: datacenter — fail (listed on a subscribed feed)
    • anomaly_score: 0.25 — pass (combined disagreement across the collected signals)
  2. signals
    • threat_feed: datacenter — fail (listed on a subscribed feed)
    • anomaly_score: 0.25 — pass (combined disagreement across the collected signals)
  3. signals
    • anomaly_score: 0.40 — pass (combined disagreement across the collected signals)

botscope.io's own traffic only — never a customer's. No addresses, no query strings, and a path is shown only when it is already a public page.

In short

  • Anything a browser can render, a determined scraper can collect. The goal is cost, not impossibility.
  • Most scraping is not determined: it is a script that stops the moment it starts failing.
  • Botscope separates the cheap majority from the persistent minority, so effort goes where it changes something.
  • Observe mode shows what a policy would have stopped before it stops anything.

Updated 16 September 2026

01 · two populations, one label

Two populations, one label

Most scraping is commodity: a library, a datacentre address, a default user agent, no session handling. It gives up as soon as responses stop being useful.

A small minority drives a real browser from residential addresses and paces itself deliberately. It is expensive to run, which is why it is aimed at content worth the expense.

Treating both as one problem produces a policy that is too aggressive for the first population and irrelevant to the second.

02 · scrapers and automation frameworks in the catalogue

Scrapers and automation frameworks in the catalogue

Bot Operator Default policy Verifiable
AI2Bot Ai2 Observe By user agent
Ai2Bot-Dolma Ai2 Observe By user agent
Applebot-Extended Apple Observe By user agent
bedrockbot Amazon Observe By user agent
Bibliotheque Nacional de France Crawler Bibliothèque nationale de France Observe By user agent
Bytespider ByteDance Observe By user agent
CCBot Common Crawl Observe By user agent
ChatGLM-Spider Zhipu AI Observe By user agent
ClaudeBot Anthropic Observe By user agent
CloudVertexBot Google Observe By user agent

Read live from the Botscope crawler catalogue. Default policy is the catalogue default; every agent can be overridden per site.

03 · what actually raises the cost

What actually raises the cost

  1. 1 Classify at the origin rather than in the page, so the check does not depend on script execution.
  2. 2 Verify declared crawlers instead of trusting the string they send.
  3. 3 Apply rate and challenge policies per agent, so a legitimate crawler is not caught by a rule aimed at somebody else.
  4. 4 Watch what changes after enforcement. A scraper that adapts is telling you it is worth adapting.

04 · what will not work

What will not work

Blocking by user agent, which the client controls. Blocking by address alone, which rotates. Obfuscating markup, which costs your own accessibility more than it costs a scraper.

And no detector separates a careful headless browser from a person with certainty. A tool claiming otherwise is describing one that also refuses people.

Questions

Can you stop scraping completely?

No, and neither can anything else. What is achievable is making it expensive enough that the commodity majority stops.

Will this block search engines?

Search crawlers are verified and allowed by default. Policies are per agent, so a rule aimed at scrapers does not apply to them.

What about false positives?

Any behavioural rule has them. Observe mode exists so you can see what a rule would have caught before it catches anything, and every decision carries a reason code you can read.

Does this need a DNS change?

No. Botscope runs at your origin; nothing is proxied through us.

See which of these reach your site

Free plan, no card: 100,000 events a month with every feature on. Nothing is blocked until you turn enforcement on.

Start free Browse the catalogue

Related