Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,953 results
Carnegie Mellon University
CMU-Cylab
Coccoc
coccocbot
coccoc
Coc Coc
coccocbot-image
Coc Coc
coccocbot-web
Cocolyze
Cocolyzebot
CodaBot
GitHub
Code
coexel
cognitive SEO Internet Marketing Tools.
cognitiveSEO
Cohere
cohere-training-data-crawler
Cohere
cohere-ai
Cohere
cohere-training-data-crawler
Cohere
CohereBot
Coinbase
weipay-webhooks
Cloudflare
Collapsify
colly
CommaFeed
Companybook-Crawler
composer
ConsentCheck
consentcheck
content
ContentAnalyzer
ContentKing
contentking
Contentsquare
ContentSquare
ContextAd
Outcomes
ContextualBot
Amazon
contxbot
convera
ConveraCrawler
Convermax
Convermax
Convermax Corp.
ConvermaxBot
CookieHub
Cookie
Cookie Maestro
cookie-maestro
Cookie Maestro
cookie-maestro
Cybot A/S
Cookiebot
CookieHub
CookieHub
CookieHub
CookieHubVerify
CookieYes
CookieYesbot
CoolBot
Copernic
CopyRightCheck
Corporama
Cosmos
Research Organization of Information and Systems
Cotoyogi
Coveo
Coveobot
OurFamilyWizard
Cozi-iCalendar-FeedReader
CragSoftware
CragCrawler
Open-source
Crawl4AI
CrawlBot
crawler_eb_germany
Freespoke
crawler.freespoke.com
crawler4j
Crawlson
Crawlson
Crawlspace
Crawlspace
CrawlyProjectCrawler
CrazyEgg
CrazyEgg
Crazy
Crescent