Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,284 results
CloudVertexBot
ai scraper
CloudVertexBot
observe
ClueWeb-Crawler
other
ClueWeb-Crawler
observe
CMS-Checker
other
CMS-Checker
observe
CMU-Cylab
ai scraper
Carnegie Mellon University
CMU-Cylab
observe
coccoc
other
coccoc
observe
CodaBot
other
CodaBot
observe
Code
ai coding agent
GitHub
Code
observe
coexel
other
coexel
observe
Cohere Training Data Crawler
ai crawler
Cohere
cohere-training-data-crawler
observe
cohere-ai
ai agent
Cohere
cohere-ai
observe
cohere-training-data-crawler
ai scraper
Cohere
cohere-training-data-crawler
observe
CohereBot
ai scraper
Cohere
CohereBot
observe
Collapsify
ai assistant
Cloudflare
Collapsify
observe
colly
other
colly
observe
CommaFeed
other
CommaFeed
observe
Companybook-Crawler
other
Companybook-Crawler
observe
Composer
ai assistant
composer
observe
signed
ConsentCheck Bot
other
ConsentCheck
consentcheck
observe
content crawler spider
other
content
observe
ContentAnalyzer
other
ContentAnalyzer
observe
ContextAd Bot
other
ContextAd
observe
contxbot
ai assistant
Amazon
contxbot
observe
convera
other
convera
observe
ConveraCrawler
other
ConveraCrawler
observe
CoolBot
other
CoolBot
observe
Copernic
other
Copernic
observe
CopyRightCheck
other
CopyRightCheck
observe
Corporama
other
Corporama
observe
Corporama matcher
other
observe
Cosmos
other
Cosmos
observe
Cotoyogi
ai crawler
Research Organization of Information and Systems
Cotoyogi
observe
Cozi-iCalendar-FeedReader
ai assistant
OurFamilyWizard
Cozi-iCalendar-FeedReader
observe
CragCrawler
ai data provider
CragSoftware
CragCrawler
observe
crawl4ai
ai crawler
Open-source
Crawl4AI
observe
CrawlBot
other
CrawlBot
observe
crawler_eb_germany
other
crawler_eb_germany
observe
crawler4j
other
crawler4j
observe
Crawlspace
ai data provider
Crawlspace
Crawlspace
observe
CrawlyProjectCrawler
other
CrawlyProjectCrawler
observe
Crazy Egg
ai coding agent
CrazyEgg
observe
Crescent
other
Crescent
observe
Crescent Internet ToolPak HTTP OLE Control v.1.0
other
observe
Critical CSS Bot
other
Critical CSS
Critical
observe
Criticalcss.com
ai coding agent
Critical CSS
Criticalcss.com
observe
CrunchBot
other
CrunchBot
observe
CrystalSemanticsBot
other
CrystalSemanticsBot
observe
Curebot
other
Curebot
observe
Current
ai assistant
Current
Current
observe
Cursor
ai coding agent
xAI
Cursor
observe
Cutbot
other
Cutbot
observe
CybaaBot
other
Cybaa
CybaaBot
observe
CyberFindCrawler
other
CyberFindCrawler
observe
CyberPatrol
other
CyberPatrol
observe
CyotekWebCopy
other
CyotekWebCopy
observe
d-bot-health-check
other
d-bot-health-check
observe
DareBoost
other
DareBoost
observe
Daric2
ai crawler
Cloudflare
daric2
observe
Daric3
ai crawler
Cloudflare
daric3
observe
Daric4
ai crawler
Cloudflare
daric4
observe
Dark Visitor
other
Dark
observe