Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,426 results
ClickUpLinkUnfurler
ai assistant
ClickUp
ClickUpLinkUnfurler
observe
Cliqzbot
other
Cliqzbot
observe
Cloudflare AutoRAG
ai crawler
Cloudflare
Cloudflare-AutoRAG
observe
Cloudflare Browser Run
ai assistant
Cloudflare
cloudflare-browser-rendering
observe
signed
Cloudflare Crawler
ai crawler
Cloudflare
Cloudflare Custom Hostname Verification
observe
signed
Cloudflare CSUP
other
Cloudflare
cloudflare-csup
observe
Cloudflare Validator
other
Cloudflare
Cloudflare-Validator
observe
Cloudflare-AI-Search-External
ai crawler
Cloudflare
Cloudflare-AI-Search-External
observe
CloudFlare-AlwaysOnline
other
CloudFlare-AlwaysOnline
observe
CloudflareBrowserRenderingCrawler
ai data provider
Cloudflare
CloudflareBrowserRenderingCrawler
observe
CloudflareWebScanner
other
CloudflareWebScanner
observe
CloudVertexBot
ai scraper
CloudVertexBot
observe
ClueWeb-Crawler
other
ClueWeb-Crawler
observe
CMS-Checker
other
CMS-Checker
observe
cms-scanner
other
cms-scanner
observe
CMS-Security-Auditor
other
CMS-Security-Auditor
observe
CMU-Cylab
ai scraper
Carnegie Mellon University
CMU-Cylab
observe
coccoc
other
coccoc
observe
CodaBot
other
CodaBot
observe
Code
ai coding agent
GitHub
Code
observe
CodeMMPretrainCrawler
other
CodeMMPretrainCrawler
observe
Codex
ai coding agent
OpenAI
Codex
observe
coexel
other
coexel
observe
Cohere Training Data Crawler
ai crawler
Cohere
cohere-training-data-crawler
observe
cohere-ai
ai agent
Cohere
cohere-ai
observe
cohere-training-data-crawler
ai scraper
Cohere
cohere-training-data-crawler
observe
CohereBot
ai scraper
Cohere
CohereBot
observe
Collapsify
ai assistant
Cloudflare
Collapsify
observe
colly
other
colly
observe
CommaFeed
other
CommaFeed
observe
Companybook-Crawler
other
Companybook-Crawler
observe
CompanyInternalSearchBot
other
CompanyInternalSearchBot
observe
CompanyMapBot
ai assistant
CompanyMapBot
observe
Composer
ai assistant
composer
observe
signed
ConsentCheck Bot
other
ConsentCheck
consentcheck
observe
content crawler spider
other
content
observe
ContentAnalyzer
other
ContentAnalyzer
observe
ContentStudio
ai assistant
ContentStudio
ContentStudio
observe
ContentStudio-Feed-Fetcher
ai assistant
ContentStudio
ContentStudio-Feed-Fetcher
observe
ContextAd Bot
other
ContextAd
observe
ContextWebCom-AdsTxt
ai assistant
ContextWebCom-AdsTxt
observe
contxbot
ai assistant
Amazon
contxbot
observe
convera
other
convera
observe
ConveraCrawler
other
ConveraCrawler
observe
CoolBot
other
CoolBot
observe
Copernic
other
Copernic
observe
CopyRightCheck
other
CopyRightCheck
observe
Corporama
other
Corporama
observe
Corporama matcher
other
observe
Cosmos
other
Cosmos
observe
Cotoyogi
ai crawler
Research Organization of Information and Systems
Cotoyogi
observe
Cozi-iCalendar-FeedReader
ai assistant
OurFamilyWizard
Cozi-iCalendar-FeedReader
observe
CragCrawler
ai data provider
CragSoftware
CragCrawler
observe
crawl4ai
ai crawler
Open-source
Crawl4AI
observe
CrawlBot
other
CrawlBot
observe
crawler_eb_germany
other
crawler_eb_germany
observe
crawler4j
other
crawler4j
observe
Crawlspace
ai data provider
Crawlspace
Crawlspace
observe
CrawlyProjectCrawler
other
CrawlyProjectCrawler
observe
Crescent
other
Crescent
observe