Skip to main content
Developer Docs

Documentation

Everything you need to integrate Botscope into your stack.

Quick Start 5 min

Create a site in the dashboard, then pick one of three connect modes under Sites → Connect. There's no Composer package — every mode is set up entirely from the dashboard.

A Via DNS — zero server changes

Point your domain's A record at our edge (the Connect tab gives you the exact records). All four protection layers run on our proxy — nothing to install, no code to touch. Works with any language or CMS.

B Via Agent — for PHP hosts you can upload to
  1. Download the bundle from Sites → Connect → Agent — it's a ZIP with agent.php, setup.php, and a config.php already filled in with your site's credentials. Nothing to edit by hand.
  2. Unpack it into an agent/ folder in your site root and upload via FTP or your host's file manager.
  3. Open https://yourdomain.com/agent/setup.php?action=install (or click "Run remote install" in the dashboard) — this wires auto_prepend_file into .htaccess / .user.ini for you. No changes to your application code.
  4. Back in the dashboard, click Verify.
C Observe JS — one script tag, no blocking

For sites where you can't touch DNS or upload anything server-side. Copy the snippet from Sites → Connect → JS only and paste it before </head>. It streams browser signals and AI-citation tracking straight to the dashboard — there's no enforcement in this mode (no blocking, no challenges), since there's no server-side component to act on a decision. Switch to DNS or Agent later for enforcement.

html 
<script src="https://botscope.io/sdk/botscope.min.js"
        data-public-token="YOUR_SITE_PUBLIC_TOKEN"
        data-panel-url="https://botscope.io"
        async></script>

data-public-token is intentionally a separate, low-privilege credential from your Site API key — safe to leave in public HTML, since it can only ever submit telemetry (never read rules, banned IPs, or config).

Want DNS's zero-load edge and a second, HMAC-verified layer of defense in case DNS is ever bypassed? That's not a fourth mode to set up — just also install the Agent (option B) on a site already running DNS (option A). Botscope enables the extra layer automatically.

You're live. DNS and Agent sync config every 5 min and keep robots.txt fresh; all three modes stream telemetry as soon as traffic arrives. Open the dashboard to see it.

Check endpoint

Called on every request. Returns a decision cached by the agent for ttl seconds.

POST /api/v1/agent/check · Request 
{
  "ua":                  "Mozilla/5.0 …",
  "ip":                  "66.249.64.0",
  "path":                "/products",
  "visitor_id":          "abc123",
  "known_bot_slug":      "googlebot",
  "web_bot_auth_status": "verified_signed",
  "robots_disallowed":   false
}
Response 
{
  "action":          "pass",
  "ttl":             300,
  "decision_reason": "known_bot:googlebot",
  "challenge": {
    "type":         "pow",
    "status_code":  403,
    "accent_color": "#6366f1"
  }
}
pass observe challenge block

Sync endpoint

Call on startup and every ~5 minutes. Returns full config including policy rules, feed blocklists, and robots.txt token list.

POST /api/v1/agent/sync → Response (excerpt) 
{
  "mode": "enforce",
  "robots_txt": {
    "disallowed_ua_tokens": ["GPTBot", "CCBot", "ByteSpider"],
    "etag": "d41d8cd9"
  },
  "rules": [ … ],
  "agent": { "auto_update": true, "latest_version": "1.4.2" }
}

disallowed_ua_tokens is the fast-path block list — bots in this list that send requests are immediately blocked with reason robots_txt_violation, even without a /check call.

robots.txt API

Fetch the generated robots.txt for your site. Supports ETag / If-None-Match to avoid unnecessary writes.

shell 
curl -s \
  -H "X-Botscope-Api-Key: $API_KEY" \
  https://app.botscope.io/api/v1/agent/robots.txt \
  > /var/www/html/robots.txt

Terminology

"Bot," "crawler," "scraper," and "agent" get used loosely across the industry. Here's exactly what each means in Botscope — the dashboard and this documentation use these terms consistently.

Bot

The umbrella term for any non-human request. Everything below is a bot — the dashboard's model is binary: a request is either a verified human or a bot.

Known Bot

A bot matched to a specific named entry in our catalog of 1,876+ bots via its UA or IP signature. Every known bot has an operator (Google, OpenAI…) and one purpose:

Crawler — Systematically visits and indexes pages for a search or discovery index — Googlebot, Bingbot, most AI search bots.

AI Agent — Acts live on behalf of a user in response to a request — answering a question, browsing a task. ChatGPT fetching a page a user asked about, an AI coding agent.

AI Scraper — Bulk-extracts content, typically to build AI training data. Always written "AI Scraper" — never bare "Scraper" — see Malicious Scraper below.

Other — Utility bots that don't fit the above: preview-card generators, uptime monitors, marketing tools.

Unidentified Bot

Non-human traffic with no catalog match — classified only by behavior, not identity:

Verified Crawler — A known crawler confirmed by rDNS, CIDR, or Web Bot Auth — genuinely who it claims to be.

Fake Crawler — Claims to be a known crawler (usually Googlebot) but fails verification — a threat, not a real crawler.

Headless Driver — Scripted or headless browser automation with no known identity.

JS-Deaf Bot — Never executes JavaScript and reports no client telemetry, with no known identity.

Malicious Scraper — Honeypot-triggered or high-rate bulk extraction with no known identity — distinct from the legitimate AI Scraper category above.

Session Thief / Token Copier — Security-abuse patterns — session replay or cookie theft, not content extraction.

One more sense of the word: Agent also refers to the installable PHP component you can deploy on your origin server — one of the three connect methods (DNS / Agent / Observe JS) covered in Quick Start (installing the Agent on top of an existing DNS site adds a second, HMAC-verified layer automatically — not a separate mode to pick). That's a deployment method, not a traffic classification — it never means "bot" anywhere in this documentation.

Identity verification signals

Distinct from the Core Engine's L1–L4 protection pipeline (see Four layers of defense) — these are the specific checks the Core Engine's L1 network layer runs to tell a real crawler from an impostor.

UA

UA + Known Bot Matching

Substring match against 1,876+ known bot patterns. Forbidden patterns disambiguate overlapping UAs.

IP

IP Verification

rDNS lookup and CIDR range check against IP ranges published by bot operators (Google, OpenAI, Anthropic, Cloudflare…).

WBA

Web Bot Auth

Cryptographic signature verification per the WICG standard. A verified signature is unforgeable.

ANOM

Behavioural Scoring

Anomaly detection: JS-deaf clients, TLS fingerprint mismatches, headless browser signals, session timing patterns.

IP verification

Bots that claim to be a trusted crawler but come from an unverified IP are treated as imposters and challenged.

Googlebot

rDNS

Bingbot

rDNS

GPTBot

CIDR

ClaudeBot

CIDR

PerplexityBot

CIDR

Applebot

rDNS

Yandexbot

rDNS

AhrefsBot

CIDR

Web Bot Auth

The WICG Web Bot Auth standard lets bots sign requests with a private key. Botscope verifies the signature against the public key directory published by the operator. A verified_signed status is the strongest possible identity signal — unforgeable without the private key.

Ready to integrate?

Check the full API Reference or start your free account.

API Reference → Start free