Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
2,114 results
changedetection
Channable
channable/importer
Channel3
Channel3Bot
ChargeBee
ChargeBee
Zhipu AI
ChatGLM-Spider
OpenAI
ChatGPT
OpenAI
ChatGPT-User
Checkly
Checkly
CheckMark Network
CheckMarkNetwork
CheckView.io
CheckView
CheeseBot
CherryPicker
CherryPickerElite
CherryPickerSE
ChimeUnfurlAgent
Binary Canary
Chirp
Chrome-Lighthouse
CINC
Cincraw
cis5550-crawler
Cision
CISPA
citeseerxbot
Citibot
CitibotSiteCrawler
Wikimedia
Citoid
seoClarity
ClarityBot
Anthropic
ClaudeBot
Anthropic
Claude-Code
Anthropic
Claude-SearchBot
Anthropic
Claude-User
Anthropic
Claude-Web
Anthropic
ClaudeBot
Mushi Labs
Clearscopebot
Cledara
CledaraBot
Cledara
Clickagy
Clickagy Intelligence Bot v2
Clickagy
Clickagy
ClickUp
ClickUpLinkUnfurler
Cliqzbot
Cloudflare
Cloudflare-AutoRAG
Cloudflare
cloudflare-browser-rendering
Cloudflare
Cloudflare Custom Hostname Verification
Cloudflare
cloudflare-csup
Cloudflare
Cloudflare
Cloudflare
CloudflareDiagnostics
Cloudflare
Cloudflare
Cloudflare-Healthchecks
Cloudflare
curly-
Cloudflare
CloudFlare-Prefetch
Cloudflare
Cloudflare
cloudflare-radar-url-scanner
Cloudflare
Cloudflare SpeedTest
Cloudflare
Cloudflare-SSLDetector
Cloudflare
Cloudflare Stream Webhook
Cloudflare
Cloudflare-Validator
Cloudflare
CloudflareWebScanner
Cloudflare
Cloudflare-AI-Search-External
CloudFlare-AlwaysOnline