PublicWWWBot
recommended: allowPublicWWWBot crawls website source code for PublicWWW. Its index lets users search HTML, JavaScript, and CSS for code snippets, technologies, analytics identifiers, and other implementation details.
What it does
PublicWWWBot follows an algorithmic schedule shaped by your site's popularity, update cadence, and domain authority. It generally spreads requests across recurring crawl passes, throttles itself to limit server load, and respects crawl rules more reliably than most other bots.
How to identify it
User-agent contains any of:
PublicWWWBot
robots.txt
Well-behaved crawlers honor robots.txt. To control PublicWWWBot:
Allow full access
User-agent: PublicWWWBot
Allow: /Block entirely
User-agent: PublicWWWBot
Disallow: /
Grey-area scrapers ignore robots.txt. Botscope enforces your policy at the edge or origin regardless —
see how →
See PublicWWWBot on your own site
Botscope shows every bot and AI agent hitting your site — and lets you allow, challenge, or block each one. Observe first, enforce when you're ready.
Start free — connect in minutes