Updated daily
Crawler Catalog
1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
812 results · Other
CherryPickerSE
other
CherryPickerSE
observe
ChimeUnfurlAgent
other
ChimeUnfurlAgent
observe
cis5550-crawler
other
cis5550-crawler
observe
Cision
other
Cision
observe
CISPA
other
CISPA
observe
citeseerxbot
other
citeseerxbot
observe
Cliqzbot
other
Cliqzbot
observe
Cloudflare CSUP
other
Cloudflare
cloudflare-csup
observe
Cloudflare Validator
other
Cloudflare
Cloudflare-Validator
observe
CloudFlare-AlwaysOnline
other
CloudFlare-AlwaysOnline
observe
ClueWeb-Crawler
other
ClueWeb-Crawler
observe
CMS-Checker
other
CMS-Checker
observe
coccoc
other
coccoc
observe
CodaBot
other
CodaBot
observe
coexel
other
coexel
observe
Coinbase Webhooks
other
Coinbase
allow
colly
other
colly
observe
CommaFeed
other
CommaFeed
observe
Companybook-Crawler
other
Companybook-Crawler
observe
ConsentCheck Bot
other
ConsentCheck
observe
content
other
content
observe
ContentAnalyzer
other
ContentAnalyzer
observe
ContextAd
other
ContextAd
observe
convera
other
convera
observe
ConveraCrawler
other
ConveraCrawler
observe
CoolBot
other
CoolBot
observe
Copernic
other
Copernic
observe
CopyRightCheck
other
CopyRightCheck
observe
Corporama
other
Corporama
observe
Cosmos
other
Cosmos
observe
CrawlBot
other
CrawlBot
observe
crawler_eb_germany
other
crawler_eb_germany
observe
crawler4j
other
crawler4j
observe
CrawlyProjectCrawler
other
CrawlyProjectCrawler
observe
Crescent
other
Crescent
observe
Critical
other
Critical CSS
Critical
observe
Critical CSS Bot
other
Critical CSS
observe
cron-job.org
other
cron-job.org
allow
CrunchBot
other
CrunchBot
observe
CrystalSemanticsBot
other
CrystalSemanticsBot
observe
Curebot
other
Curebot
observe
Cutbot
other
Cutbot
observe
CybaaBot
other
Cybaa
CybaaBot
observe
CyberFindCrawler
other
CyberFindCrawler
observe
CyberPatrol
other
CyberPatrol
observe
CyotekWebCopy
other
CyotekWebCopy
observe
d-bot-health-check
other
d-bot-health-check
observe
DareBoost
other
DareBoost
observe
Dark
other
Dark
observe
Datafeedwatch
other
Datafeedwatch
observe
datagnionbot
other
datagnionbot
observe
Datanyze
other
Datanyze
observe
Daumoa
other
Daumoa
observe
Dazzle
other
Dazzle
observe
dbot
other
dbot
observe
deepnoc
other
deepnoc
observe
DeuSu
other
DeuSu
observe
Digg
other
Digg
observe
Digimind
other
Digimind
observe
Digincore
other
Digincore
observe