Verifying Googlebot: rDNS vs CIDR vs Web Bot Auth

Why the User-Agent header isn't enough

Any HTTP client can send any User-Agent string it wants. This works today, with no special tools:

curl -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://example.com

That request will show up in your logs looking exactly like Googlebot, because the header is just a piece of text the client chose to send — there's nothing in HTTP that verifies it. A malicious scraper that wants to bypass a rule like "allow Googlebot, block everything else" only has to copy this one string. If your allowlist trusts the header alone, it isn't actually a Googlebot allowlist; it's an anyone-who-read-your-source allowlist.

There are three real ways to confirm a crawler is who it claims to be. Each has a genuine trade-off in latency, accuracy, and how much work it is to maintain.

Method 1: Reverse DNS + forward confirmation

This is Google's own documented method for verifying Googlebot, and it works in two steps:

1. Reverse-resolve the visiting IP to a hostname (a PTR lookup). A genuine Googlebot request resolves to a hostname ending in .googlebot.com or .google.com.
2. Forward-resolve that hostname back to an IP (an A/AAAA lookup) and confirm it matches the original visiting IP. This second step is what makes it forward-confirmed reverse DNS (FCrDNS) — without it, an attacker who controls reverse DNS for their own IP range (which anyone renting a server typically can configure) could simply set their PTR record to something ending in googlebot.com and pass a check that only did step 1.

# Step 1: reverse lookup
$ dig -x 66.249.66.1 +short
crawl-66-249-66-1.googlebot.com.

# Step 2: forward lookup on the returned hostname — must match the original IP
$ dig crawl-66-249-66-1.googlebot.com +short
66.249.66.1

Pros:

• Free, and officially documented by Google — no third-party dependency.
• Works today with zero setup beyond DNS lookups you already have the tools for.
• Doesn't require you to track and refresh a list of anything.

Cons:

• Two DNS lookups per uncached request adds real latency — you'll want to cache results by IP for a sensible TTL rather than looking up on every hit.
• Only works for crawlers that actually maintain accurate reverse DNS records — plenty of smaller or less careful bot operators don't.

Method 2: Published IP ranges (CIDR)

Google (and Bing, and several other major crawler operators) publish a JSON list of the IP ranges their crawlers use. Verification becomes a simple check: is the visiting IP inside one of the published CIDR blocks?

// Example published range entry
{ "ipv4Prefix": "66.249.64.0/19" }

// Verification is just a CIDR containment check
ip_in_cidr("66.249.66.1", "66.249.64.0/19") // true

Pros:

• Fast — a single containment check, no DNS round-trip, negligible latency.
• Simple to implement and simple to cache; the published lists change infrequently.

Cons:

• You must keep your local copy of the list current — operators do rotate and add ranges over time.
• Coarse-grained by nature: it proves the request came from an IP the operator has claimed, not that this specific request is legitimate crawler traffic versus something else sharing that infrastructure.
• You're trusting the delivery channel for the list itself — fetch it only from the operator's official, HTTPS-verified source.

Method 3: Web Bot Auth (cryptographic signatures)

Web Bot Auth is the newest approach, built on IETF HTTP Message Signatures. Instead of relying on network-level facts (which IP, which hostname), the crawler operator signs every request with a private key, and your server verifies that signature against a public key the operator publishes at a well-known directory URL.

Signature-Input: sig1=("@method" "@path" "host");keyid="poiuytrewq";alg="ed25519"
Signature: sig1=:MEUCIQDx7...base64...==:

Pros:

• Cryptographically strong — spoofing it means forging a signature, not just copying a header or an IP.
• No dependency on DNS or IP at all — works correctly even behind CDNs and proxies where the visible source IP isn't the crawler's own.
• Extensible to any operator willing to sign requests, not just search engines — a growing number of AI agents are adopting it too.

Cons:

• Still early — adoption is limited to operators who've implemented signing, which today is a smaller set than those with documented IP ranges.
• More implementation work on your side: fetching and caching the operator's key directory, and validating a signature per request rather than a simple lookup.

Comparing all three

Which should you actually use?

In practice, the strongest setup is layered rather than a single method:

1. CIDR as the fast first-pass filter for the small set of major, well-documented crawlers — it's cheap and catches the overwhelming majority of legitimate traffic instantly.
2. Forward-confirmed rDNS as the authoritative fallback for anything not in a cached range, or as a second opinion when you want higher confidence before granting access to something sensitive.
3. Web Bot Auth wherever the operator supports it, since it's strictly the strongest guarantee and requires no DNS or IP-list trust at all — treat it as the long-term default as adoption grows.
4. TLS fingerprint verification as a cross-check against any of the above — a request that clears an IP or DNS check but negotiates TLS like a script rather than a real crawler is still worth a second look; see our JA3/JA4 fingerprinting piece for how that works.

This is exactly the verification chain Botscope runs on every request that claims to be a known crawler: CIDR first for speed, forward-confirmed rDNS as the fallback, and Web Bot Auth signature verification wherever the operator publishes one — so a spoofed Googlebot User-Agent gets caught before it ever reaches your application. See the full list of crawlers we verify in the crawler catalog.

FAQ

Is checking the User-Agent string ever useful?

It's useful as a first-pass hint to decide which verification method to run — if a request claims to be Googlebot, check it against Google's published ranges or rDNS suffix. It should never be the check itself, since it's trivially forged.

Do I need to verify every single request?

No — cache verification results by IP (for CIDR/rDNS) or by key ID (for Web Bot Auth) for a reasonable TTL. Crawler IP ranges and signing keys don't change every request, so re-verifying on every single hit wastes latency for no additional confidence.

What happens if a request fails verification?

That's a policy decision, not a technical one — you might block it outright, silently serve a degraded experience, or route it to a challenge. Botscope treats a failed verification as "unverified automation" rather than "confirmed bad," so you can choose how strictly to enforce it per site.