Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
908 results · Other
CDSCbot
other
CDSCbot
observe
Cegbfeieh
other
Cegbfeieh
observe
CentComBot
other
MelonNET
CentComBot
observe
Centro Ads.txt Crawler
other
Centro
observe
Centurybot
other
Centurybot
observe
changedetection
other
changedetection
observe
ChargeBee
other
ChargeBee
ChargeBee
allow
CheeseBot
other
CheeseBot
observe
CherryPicker
other
CherryPicker
observe
CherryPickerElite
other
CherryPickerElite
observe
CherryPickerSE
other
CherryPickerSE
observe
ChimeUnfurlAgent
other
ChimeUnfurlAgent
observe
cis5550-crawler
other
cis5550-crawler
observe
Cision
other
Cision
observe
CISPA Webcrawler
other
CISPA
observe
citeseerxbot
other
citeseerxbot
observe
Cliqzbot
other
Cliqzbot
observe
Cloudflare CSUP
other
Cloudflare
cloudflare-csup
observe
Cloudflare Validator
other
Cloudflare
Cloudflare-Validator
observe
CloudFlare-AlwaysOnline
other
CloudFlare-AlwaysOnline
observe
CloudflareWebScanner
other
CloudflareWebScanner
observe
ClueWeb-Crawler
other
ClueWeb-Crawler
observe
CMS-Checker
other
CMS-Checker
observe
cms-scanner
other
cms-scanner
observe
CMS-Security-Auditor
other
CMS-Security-Auditor
observe
coccoc
other
coccoc
observe
CodaBot
other
CodaBot
observe
CodeMMPretrainCrawler
other
CodeMMPretrainCrawler
observe
coexel
other
coexel
observe
Coinbase Webhooks
other
Coinbase
weipay-webhooks
allow
colly
other
colly
observe
CommaFeed
other
CommaFeed
observe
Companybook-Crawler
other
Companybook-Crawler
observe
CompanyInternalSearchBot
other
CompanyInternalSearchBot
observe
ConsentCheck Bot
other
ConsentCheck
consentcheck
observe
content crawler spider
other
content
observe
ContentAnalyzer
other
ContentAnalyzer
observe
ContextAd Bot
other
ContextAd
observe
convera
other
convera
observe
ConveraCrawler
other
ConveraCrawler
observe
CoolBot
other
CoolBot
observe
Copernic
other
Copernic
observe
CopyRightCheck
other
CopyRightCheck
observe
Corporama
other
Corporama
observe
Corporama matcher
other
observe
Cosmos
other
Cosmos
observe
CrawlBot
other
CrawlBot
observe
crawler_eb_germany
other
crawler_eb_germany
observe
crawler4j
other
crawler4j
observe
CrawlyProjectCrawler
other
CrawlyProjectCrawler
observe
Crescent
other
Crescent
observe
Crescent Internet ToolPak HTTP OLE Control v.1.0
other
observe
Critical CSS Bot
other
Critical CSS
Critical
observe
cron-job.org
other
cron-job.org
cron-job.org
allow
CrunchBot
other
CrunchBot
observe
CrystalSemanticsBot
other
CrystalSemanticsBot
observe
CT-WP-Probe
other
CT-WP-Probe
observe
Curebot
other
Curebot
observe
Cutbot
other
Cutbot
observe
CybaaBot
other
Cybaa
CybaaBot
observe