Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
2,114 results
Cloudflare
Cloudflare-Traffic-Manager
Cloudflare
CloudflareBrowserRenderingCrawler
CloudflareWebScanner
Cloudtrellis
Cloudtrellis
CloudVertexBot
Cludo
cludo.com
Cludo
cludo.com
ClueWeb-Crawler
CMS-Checker
cms-scanner
CMS-Security-Auditor
Carnegie Mellon University
CMU-Cylab
Coccoc
coccocbot
coccoc
Coc Coc
coccocbot-image
Coc Coc
coccocbot-web
Cocolyze
Cocolyzebot
CodaBot
GitHub
Code
CodeMMPretrainCrawler
OpenAI
Codex
coexel
cognitive SEO Internet Marketing Tools.
cognitiveSEO
Cohere
cohere-training-data-crawler
Cohere
cohere-ai
Cohere
cohere-training-data-crawler
Cohere
CohereBot
Coinbase
weipay-webhooks
Cloudflare
Collapsify
colly
CommaFeed
Companybook-Crawler
CompanyInternalSearchBot
CompanyMapBot
composer
ConsentCheck
consentcheck
content
ContentAnalyzer
ContentKing
contentking
Contentsquare
ContentSquare
ContentStudio
ContentStudio
ContentStudio
ContentStudio-Feed-Fetcher
ContextAd
Outcomes
ContextualBot
ContextWebCom-AdsTxt
Amazon
contxbot
convera
ConveraCrawler
Convermax
Convermax
Convermax Corp.
ConvermaxBot
CookieHub
Cookie
Cookie Maestro
cookie-maestro
Cybot A/S
Cookiebot
CookieHub
CookieHubVerify
CookieYes
CookieYesbot
CoolBot
Copernic
CopyRightCheck
Corporama