Google AdSense
recommended: observeGoogle AdSense is an intelligence gatherer operated by Google.
What it does
Google AdSense usually returns to a small set of pages rather than crawling an entire site. Visits may follow an hourly, daily, or weekly schedule, while websites its clients do not monitor may never encounter it.
How to identify it
User-agent contains any of:
Google AdSense
Example user-agent string:
Mediapartners-GoogleHow to verify it's genuine
Reverse-DNS the visiting IP; a genuine request resolves to a hostname ending in:
rate-limited-proxy-***-***-***-***.google.com
Then forward-resolve that hostname back to the same IP (forward-confirmed rDNS) to defeat spoofing.
robots.txt
Well-behaved crawlers honor robots.txt. To control Google AdSense:
Allow full access
User-agent: Google AdSense
Allow: /Block entirely
User-agent: Google AdSense
Disallow: /
Grey-area scrapers ignore robots.txt. Botscope enforces your policy at the edge or origin regardless —
see how →
See Google AdSense on your own site
Botscope shows every bot and AI agent hitting your site — and lets you allow, challenge, or block each one. Observe first, enforce when you're ready.
Start free — connect in minutes