Skip to main content

Solutions

Fake Googlebot detection

Anyone can send Googlebot’s user agent, so allowlisting that string hands attackers a bypass. How reverse DNS and published ranges actually verify a crawler.

Start free Browse the catalogue

Free plan, no card. 100,000 events a month with every feature on.

botscope.io · last 7d

This site, classified by this product. Nothing here is a mock-up.

live
requests
4,245
human
80%
AI agents
0%
stopped
0%
verdict stream
  1. signals
    • threat_feed: datacenter — fail (listed on a subscribed feed)
    • anomaly_score: 0.25 — pass (combined disagreement across the collected signals)
  2. signals
    • threat_feed: datacenter — fail (listed on a subscribed feed)
    • anomaly_score: 0.25 — pass (combined disagreement across the collected signals)
  3. signals
    • anomaly_score: 0.40 — pass (combined disagreement across the collected signals)

botscope.io's own traffic only — never a customer's. No addresses, no query strings, and a path is shown only when it is already a public page.

In short

  • The Googlebot user agent is a public string with no secret in it.
  • A rule that allows traffic based on that string is a rule anyone can satisfy.
  • Verification is reverse DNS to a Google domain and forward again, or a match against published address ranges.
  • Botscope verifies before applying an allow, so an impostor does not inherit the crawler’s permission.

Updated 16 September 2026

01 · why an allowlist becomes a bypass

Why an allowlist becomes a bypass

Most sites allow search crawlers generously — skip the rate limit, skip the challenge, serve the full page. That is correct, and it depends entirely on the allow being scoped to the actual crawler.

When the scope is a user agent string, the rule reads: anyone who sets this header gets the crawler's privileges. The more carefully a site protects itself elsewhere, the more valuable that bypass becomes.

02 · how verification works

How verification works

  1. 1 Take the connecting address and resolve it backwards to a hostname.
  2. 2 Check the hostname belongs to the operator’s domain — googlebot.com or google.com for Googlebot.
  3. 3 Resolve that hostname forward again and confirm it returns the address you started with. Without this step, reverse DNS alone can be claimed by whoever controls the record.
  4. 4 Where an operator publishes address ranges or signs its requests, use that instead: it is cheaper and does not depend on DNS.

03 · crawlers with a documented verification method

Crawlers with a documented verification method

Bot Operator Default policy Verifiable
360Spider Qihoo 360 Allow By user agent
Alexa Archive Alexandria.org Allow By user agent
alexa site audit Amazon Allow By user agent
AlexandriaOrgBot Alexandria.org Allow By user agent
Algolia Algolia Allow By user agent
All Africa Crawler AllAfrica Global Media Allow By user agent
Atom Feed Robot RSSMicro Allow By user agent
Baidu ADS Server Proxy Baidu Allow By user agent
Baiduspider Baidu Allow By user agent
Baiduspider-render Baidu Allow By user agent

Read live from the Botscope crawler catalogue. Default policy is the catalogue default; every agent can be overridden per site.

04 · the cost

The cost

Reverse DNS is a network round trip, which is why sites skip it. Results are cached, and address ranges and request signatures avoid the lookup entirely where the operator supports them.

Questions

Why is the forward lookup necessary?

Reverse DNS records are controlled by whoever holds the address block, so a reverse record alone can claim any name. Resolving forward and matching the original address is what makes the pair hard to forge.

Does this slow down real crawlers?

The first request from an address pays the lookup; the result is cached. Operators that publish ranges or sign requests skip it entirely.

What about Bingbot and the AI crawlers?

Same approach, different domains and ranges. The catalogue records the applicable method for each agent.

Can I see impostors that already reached me?

Requests claiming a verified crawler that fail verification are recorded and can be reviewed, in observe mode as well as under enforcement.

See which of these reach your site

Free plan, no card: 100,000 events a month with every feature on. Nothing is blocked until you turn enforcement on.

Start free Browse the catalogue

Related