Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
844 results · Other
CheeseBot
other
CheeseBot
observe
CherryPicker
other
CherryPicker
observe
CherryPickerElite
other
CherryPickerElite
observe
CherryPickerSE
other
CherryPickerSE
observe
ChimeUnfurlAgent
other
ChimeUnfurlAgent
observe
cis5550-crawler
other
cis5550-crawler
observe
Cision
other
Cision
observe
CISPA Webcrawler
other
CISPA
observe
citeseerxbot
other
citeseerxbot
observe
Clarity-Bot
other
Clarity-Bot
observe
Cliqzbot
other
Cliqzbot
observe
Cloudflare CSUP
other
Cloudflare
cloudflare-csup
observe
Cloudflare Validator
other
Cloudflare
Cloudflare-Validator
observe
CloudFlare-AlwaysOnline
other
CloudFlare-AlwaysOnline
observe
cloudflare-csup
other
cloudflare-csup
observe
CloudflareDiagnostics
other
CloudflareDiagnostics
observe
ClueWeb-Crawler
other
ClueWeb-Crawler
observe
CMS-Checker
other
CMS-Checker
observe
coccoc
other
coccoc
observe
CodaBot
other
CodaBot
observe
coexel
other
coexel
observe
Coinbase Webhooks
other
Coinbase
weipay-webhooks
allow
colly
other
colly
observe
CommaFeed
other
CommaFeed
observe
Companybook-Crawler
other
Companybook-Crawler
observe
ConsentCheck Bot
other
ConsentCheck
consentcheck
observe
content crawler spider
other
content
observe
ContentAnalyzer
other
ContentAnalyzer
observe
ContextAd Bot
other
ContextAd
observe
convera
other
convera
observe
ConveraCrawler
other
ConveraCrawler
observe
CoolBot
other
CoolBot
observe
Copernic
other
Copernic
observe
CopyRightCheck
other
CopyRightCheck
observe
Corporama
other
Corporama
observe
Corporama matcher
other
observe
Cosmos
other
Cosmos
observe
CrawlBot
other
CrawlBot
observe
crawler_eb_germany
other
crawler_eb_germany
observe
crawler4j
other
crawler4j
observe
CrawlyProjectCrawler
other
CrawlyProjectCrawler
observe
Crescent
other
Crescent
observe
Crescent Internet ToolPak HTTP OLE Control v.1.0
other
observe
Critical CSS Bot
other
Critical CSS
Critical
observe
cron-job.org
other
cron-job.org
cron-job.org
allow
CrunchBot
other
CrunchBot
observe
CrystalSemanticsBot
other
CrystalSemanticsBot
observe
Curebot
other
Curebot
observe
Cutbot
other
Cutbot
observe
CybaaBot
other
Cybaa
CybaaBot
observe
CyberFindCrawler
other
CyberFindCrawler
observe
CyberPatrol
other
CyberPatrol
observe
CyotekWebCopy
other
CyotekWebCopy
observe
d-bot-health-check
other
d-bot-health-check
observe
DareBoost
other
DareBoost
observe
Dark Visitor
other
Dark
observe
Datafeedwatch
other
Datafeedwatch
observe
datagnionbot
other
datagnionbot
observe
Datanyze
other
Datanyze
observe
Daumoa
other
Daumoa
observe