Updated 16 September 2026
01 · what it stops
What it stops
Every item here is a decision Botscope makes per request, with the signals behind it recorded and readable afterwards.
- Content and price scraping
- Fake Googlebot and spoofed crawlers
- AI scrapers ignoring robots.txt
- Invalid traffic that inflates ad spend
- Headless browsers — Puppeteer, Playwright, Selenium
- Credential stuffing and account takeover attempts
- Fake signups and spam registrations
- Datacenter IPs, residential proxies and Tor exits
02 · what bot detection software actually needs to do
What bot detection software actually needs to do
A score between 0 and 100 tells you nothing about which signal produced it, who sent the request, or whether blocking it breaks an integration you depend on. Botscope answers with a named classification and the signals behind it, so you can check the decision instead of trusting it.
03 · why user-agent matching is not enough
Why user-agent matching is not enough
The User-Agent header is a string the requester sets. A scraper copies Googlebot's in one line of code, so the comparison has to be between what a request declares and what it does — what it fetches, in what order, from whose network, and whether it honours robots.txt. Disagreement between the two is the finding, and a user-agent list can never produce it.
Verification is four mechanisms. Of 1,953 catalogued agents, 37 are checked by reverse and forward DNS, 48 against published address ranges, 51 by Web Bot Auth signatures, and the other 1,817 on behaviour alone. The record says which of the four applied.
04 · under the hood
Four layers of defense
Every request runs the full stack before your application is reached — no redirect, no round trip, nothing for a real visitor to wait on.
Network & Crawlers
- ·IP reputation & threat feeds
- ·Verified crawler bypass (Googlebot, GPTBot…)
- ·Fake-crawler detection via rDNS mismatch
Session Validation
- ·HMAC-signed shield session cookie
- ·TLS / JA4 fingerprint continuity check
- ·Silent re-vouch on IP change (150 ms)
Challenge Barrier
- ·Invisible Proof-of-Work (browser-only)
- ·Adaptive difficulty by risk score 0–100
- ·Custom rule engine — block / challenge / observe
Behavioural Traps
- ·Invisible honeypot links in HTML
- ·Stateless pixel tracks static-asset loading
- ·Token-copier & HTML-only scraper detection
05 · the four classifications, and the evidence behind each
The four classifications, and the evidence behind each
Botscope assigns each request to one of four top-level categories. Each has a different implication for what you might want to do about it.
The evidence is the output rather than a label attached afterwards: the declared user agent, the result of whatever identity check was available, the behaviour observed, and the operator where one is known. Which matters the day an assistant you depend on turns up classified as a scraper. You read why, and then reclassify it, or write to its operator, or block it. You are not appealing a number.
- Human: A request consistent with a person using a browser. No automated fingerprints detected.
- Verified search crawler: A request from a known search engine crawler (Googlebot, Bingbot, and others in the catalogue) where identity checks pass.
- AI agent: A request from an AI system — an AI crawler, AI assistant, AI coding agent, or similar — operating within its declared scope.
- Scraper: A request that presents automated characteristics without a verifiable, legitimate identity, or that contradicts its declared identity through behaviour.
Every record above comes from a real site. Connect yours and the first ones appear within minutes.
See your own traffic07 · observation mode: classify before you commit to blocking
Observation mode: classify before you commit to blocking
Every request classified and logged, nothing blocked, challenged or rate-limited. It is where most deployments should start, because you cannot sensibly block a category before you have seen what is in it — how much of your traffic is AI crawlers, which operators send it, and whether anything you rely on is landing in the wrong bucket. Run it as long as you like.
08 · what the catalogue actually holds
What the catalogue actually holds
| Bot | Operator | Default policy | Verifiable |
|---|---|---|---|
| AI Search | Cloudflare | Observe | By user agent |
| AIWebIndex-Agent | Lyrenth | Observe | By user agent |
| Amazon Kendra | Amazon | Observe | By user agent |
| Amazonbot | Amazon | Observe | By user agent |
| Amzn-SearchBot | Amazon | Observe | By user agent |
| Anchor Browser | Anchor | Observe | Signed requests |
| Anomura | Direqt | Observe | By user agent |
| Applebot | Apple | Observe | By user agent |
| Atlassian (Rovo AI) | Atlassian | Allow | By user agent |
| atlassian-bot | Atlassian | Observe | By user agent |
| AwarioSmartBot | Awario | Observe | By user agent |
| AzureAI-SearchBot | Microsoft | Observe | By user agent |
Read live from the Botscope crawler catalogue. Default policy is the catalogue default; every agent can be overridden per site.
10 · scope and limits
Scope and limits
Not a WAF, not DDoS protection, no CAPTCHA. It classifies; it sits alongside your edge rather than replacing it.
An opaque score is the better buy when you want a number, a threshold and a rule, and nobody is going to audit any single decision. Evidence is only worth its cost when somebody reads it.
A scraper written this morning has no catalogue entry, and we will say so rather than invent a name. The behavioural gates still catch it — honeypots, missing JS telemetry, static-asset patterns, session continuity. What is lost is the operator name, not the verdict.
11 · what it costs
What it costs
The price is on this page, not after a call. Blocked and challenged requests do count toward the monthly total: a decision still produces a stored record, and that record is what an event is. Only browser engagement pings — scroll depth, time on page — are excluded.
| Events a month | Per 100K block |
|---|---|
| 100K – 600K | $5.99 |
| 600K – 2.1M | $3.99 |
| 2.1M+ | $2.99 |
Free to 100,000 events a month, no card, every feature on. Nothing is blocked until you turn enforcement on.
Start free13 · compared to the alternatives
Compared to the alternatives
Where each of these is the better buy, including where Botscope is not.
| Botscope | Cloudflare bot products | DataDome | Arcjet | |
|---|---|---|---|---|
| Starting price | Best here: Free to 100K events a month, every feature on | Bot Fight Mode is free but not configurable; Bot Management is an Enterprise add-on | No public price list; entry tier reported around $3,830/mo | 15-day trial, then from $25/mo per app plus usage |
| Time to first data | Minutes — a script tag, DNS, a PHP agent, a WordPress plugin or a Worker | Best here: Immediate if you are already on Cloudflare | A free trial, after deploying one of their integrations | A deploy — it is an SDK in your code |
| What one verdict shows you | Best here: Every signal behind it, with its value and result | A bot score from 1 to 99 and the rule that matched (Enterprise) | A score that decides allow, monitor, verify or block | The decision and its reason, returned to your code |
| Can a blocked visitor find out why | Best here: Yes — a decision ID on the block page, explained at /why-blocked | A Ray ID to give the site owner, who can look up the rule | A device check or CAPTCHA; the page does not say why | Whatever your application chooses to show |
| Checks a crawler instead of trusting its name | Yes — published ranges and forward-confirmed reverse DNS | Yes — signatures, published IP lists and reverse DNS | Yes — two-way reverse DNS, IP lists and AS numbers | Yes — checks a claimed crawler’s IP against its operator |
| Watch before anything is blocked | Observe mode, the default for every new site | A Log action on Enterprise; Bot Fight Mode has no log-only setting | Labels traffic without blocking during the trial and onboarding | DRY_RUN mode logs what it would have done |
| What it does not do | Not a WAF, not DDoS protection, no CAPTCHA | Not origin-level: it sees what reaches the edge | No public prices; plans are quoted by sales | Nothing outside the app it is installed in |
| When to pick them instead | — | You want one vendor for CDN, WAF and bots, and do not need the reason | You need a 24/7 response team and a contractual SLA | You want decisions inside your application code, not in front of it |
Questions
I already have Cloudflare Bot Fight Mode for free. Why would I add this?
For blocking known AI crawlers at the edge, you may not need to — Cloudflare does that on every plan now, and doing it twice buys nothing. What it does not give you is the reason. A request is challenged or it is not, and neither you nor the person it happened to can find out which signals decided it. Botscope classifies at your origin, records the evidence per request, and hands the blocked visitor an ID they can look up. "How do I stop GPTBot" is Cloudflare's question. "Why was this customer stopped" is not.
Will this block Googlebot or hurt my SEO?
Googlebot is verified by DNS and against the ranges Google publishes, not by its user-agent — so the real one passes and something claiming to be it from an Amazon address does not. Search crawlers are allowed by default, and nothing is enforced at all until you switch it on.
What happens to my uptime monitors, payment webhooks and Bing Site Scan?
Catalogued as the automated clients they are, and allowed by default. Anything of yours that is not — an internal script, an uncatalogued vendor — shows up in observation mode, where you can allow it before enforcement is ever on.
How do I reverse a wrong verdict, and what does the blocked visitor see?
The blocked page carries a decision ID. The visitor looks it up at /why-blocked and sees which check stopped them; you find the same request by that ID and change the agent's policy, which applies within minutes. No ticket to us involved.
What happens if Botscope is unreachable — does my site go down?
Traffic passes. The agent fails open: if the panel is unreachable it stops classifying rather than holding requests, so an outage here costs you telemetry and never availability.
Can a scraper just copy a known bot's user-agent and get classified as legitimate?
User-agent is one input among several, and never the deciding one. Botscope compares declared identity against observed behaviour, including fetch patterns, timing, header ordering, and network properties. Where those disagree, the disagreement is flagged in the evidence record. For the agents that publish DNS records, address ranges or a signing key — 37, 48 and 51 of them respectively — identity is checked without reference to the user-agent string at all.
How long should I run observation mode before enabling enforcement?
There is no fixed answer. The purpose of observation mode is to give you enough data to make an informed enforcement decision. In practice, you want to see enough traffic to identify which operators are present, whether any legitimate integrations appear in unexpected categories, and what volume each category represents. That may take days or weeks depending on your traffic patterns. Botscope does not time-limit observation mode.
Is Botscope suitable if I already have a CDN that offers bot protection?
Yes. CDN bot products and Botscope address different questions. A CDN product typically applies threshold rules at the network edge based on volumetric and network-layer signals. Botscope provides per-request classification with named operators and visible evidence. Many deployments use both: the CDN handles volumetric filtering and Botscope handles classification of the traffic that passes through.
See which of these reach your site
Free plan, no card: 100,000 events a month with every feature on. Nothing is blocked until you turn enforcement on.