Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
471 results
Chief Tools
CertChief
Chief
CertChief
Qualys
cf-qualys-scanner
ChargeBee
ChargeBee
Checkly
Checkly
CheckView.io
CheckView
Cloudflare
Cloudflare
CloudflareDiagnostics
Cloudflare
curly-
Cloudflare
cloudflare-radar-url-scanner
Cloudflare
CloudflareWebScanner
Cloudtrellis
Cloudtrellis
Cludo
cludo.com
Cludo
cludo.com
Coccoc
coccocbot
Coc Coc
coccocbot-image
Coc Coc
coccocbot-web
Coinbase
weipay-webhooks
ContentKing
contentking
Contentsquare
ContentSquare
Convermax Corp.
ConvermaxBot
CookieHub
Cookie
Cookie Maestro
cookie-maestro
Cookie Maestro
cookie-maestro
Cybot A/S
Cookiebot
CookieHub
CookieHub
CookieHub
CookieHubVerify
CookieYes
CookieYesbot
Coveo
Coveobot
Freespoke
crawler.freespoke.com
Crawlson
Crawlson
CrazyEgg
Crazy
cron-job.org
cron-job.org
C U Cyber History
CUCH.org
CybaaBot
CyberOK
CyberOKInspect
Cypex
cypex.ai/scanning
Dataprovider.com
Dataprovider
Dataprovider.com
Dataprovider
Daum
Daum
DLC Websites
deadlinkchecker
Lumar
deepcrawl
Detectify
Detectify
Discord, Inc.
Discordbot
DocSpot
DocSpotBot
Dotcom-Monitor
DMBrowser
Dr. Link Check
Drata
Dratabot
Drata
Dratabot
DuckDuckGo
DuckDuckBot
DuckDuckGo
DuckDuckGo-Favicons-Bot
EasyCron
EasyDNS
easyDNS
elmah.io
elmahio-uptimebot
Embedly
Embedly
Sebastian Foth - Software Solutions
eRepublik.tools
Evo Agency Ltd.
EvoUptimeBot
Exodus
ExodusMovement
Palo Alto Networks
Expanse
Meta
facebook