Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
497 results
Cloudflare
bushbaby
Buttondown
Buttondown RSS-Feed-Parser
Catchpoint
Catchpoint
Chief Tools
CertChief
Qualys
cf-qualys-scanner
CGMIMM
CGMSearchBot
ChargeBee
ChargeBee
Checkly
Checkly
CheckView.io
CheckView
Cloudflare
Cloudflare
CloudflareDiagnostics
Cloudflare
curly-
Cloudflare
CloudFlare-Prefetch
Cloudflare
cloudflare-radar-url-scanner
Cloudflare
Cloudflare-SSLDetector
Cloudflare
CloudflareWebScanner
Cloudflare
Cloudflare-Traffic-Manager
Cloudtrellis
Cloudtrellis
Cludo
cludo.com
Cludo
cludo.com
Coccoc
coccocbot
Coc Coc
coccocbot-image
Coc Coc
coccocbot-web
Coinbase
weipay-webhooks
ContentKing
contentking
Contentsquare
ContentSquare
Convermax Corp.
ConvermaxBot
CookieHub
Cookie
Cookie Maestro
cookie-maestro
Cybot A/S
Cookiebot
CookieHub
CookieHubVerify
CookieYes
CookieYesbot
Coveo
Coveobot
Freespoke
crawler.freespoke.com
Crawlson
Crawlson
CrazyEgg
Crazy
cron-job.org
cron-job.org
C U Cyber History
CUCH.org
CybaaBot
Cybaa
CybaaAgent
CyberConvoy
CyberConvoyScout
CyberOK
CyberOKInspect
Cypex
cypex.ai/scanning
Dataprovider.com
Dataprovider
Dataprovider.com
Dataprovider
Daum
Daum
DLC Websites
deadlinkchecker
Lumar
deepcrawl
Detectify
Detectify
Discord, Inc.
Discordbot
DocSpot
DocSpotBot
Dotcom-Monitor
DMBrowser
Pragmatic Security
DomainScores
Dr. Link Check
Drata
Dratabot
Drata
Dratabot
DuckDuckGo
DuckDuckBot
DuckDuckGo
DuckDuckGo-Favicons-Bot
EasyCron
EasyDNS
easyDNS