Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,284 results
BunnySlippers
other
BunnySlippers
observe
Buttondown
ai assistant
Buttondown
Buttondown
observe
buzzbot
other
buzzbot
observe
Bytespider
ai scraper
ByteDance
Bytespider
observe
CandidateCrawler
other
CandidateCrawler
observe
CapsuleChecker
other
CapsuleChecker
observe
CapterraBot
other
CapterraBot
observe
captify-crawler
other
Captify
captify-crawler
observe
carbon-umbrella-bot
other
carbon-umbrella-bot
observe
careerbot
other
careerbot
observe
CartAI Agentic Commerce as a Service
ai assistant
CartAI
cartai-agentic-commerce-as-a-service
observe
signed
caveman-hunter
other
caveman-hunter
observe
CC Metadata Scaper
other
observe
cc_bot
other
cc_bot
observe
CCBot
ai scraper
Common Crawl
CCBot
observe
rDNS
CDSCbot
other
CDSCbot
observe
Cegbfeieh
other
Cegbfeieh
observe
CensysInspect
security
Censys
CensysInspect
observe
CentComBot
other
MelonNET
CentComBot
observe
Centro Ads.txt Crawler
other
Centro
observe
Centurybot
other
Centurybot
observe
changedetection
other
changedetection
observe
channable
ai assistant
Channable
channable
observe
ChatGLM-Spider
ai scraper
Zhipu AI
ChatGLM-Spider
observe
ChatGPT agent
ai assistant
OpenAI
ChatGPT
observe
signed
ChatGPT-User
ai assistant
OpenAI
ChatGPT-User
observe
CheckMarkNetwork
ai assistant
CheckMark Network
CheckMarkNetwork
observe
CheeseBot
other
CheeseBot
observe
CherryPicker
other
CherryPicker
observe
CherryPickerElite
other
CherryPickerElite
observe
CherryPickerSE
other
CherryPickerSE
observe
ChimeUnfurlAgent
other
ChimeUnfurlAgent
observe
Chirp
ai coding agent
Binary Canary
Chirp
observe
Cincraw
ai assistant
CINC
Cincraw
observe
cis5550-crawler
other
cis5550-crawler
observe
Cision
other
Cision
observe
CISPA Webcrawler
other
CISPA
observe
citeseerxbot
other
citeseerxbot
observe
CitibotSiteCrawler
ai crawler
Citibot
CitibotSiteCrawler
observe
Citoid
ai assistant
Wikimedia
Citoid
observe
Clarity-Bot
other
Clarity-Bot
observe
Claude
ai crawler
Anthropic
ClaudeBot
observe
Claude-Code
ai coding agent
Anthropic
Claude-Code
observe
Claude-SearchBot
ai crawler
Anthropic
Claude-SearchBot
observe
Claude-User
ai assistant
Anthropic
Claude-User
observe
Claude-Web
ai agent
Anthropic
Claude-Web
observe
ClaudeBot
ai scraper
Anthropic
ClaudeBot
observe
Cledara SaaS Management Agent
ai assistant
Cledara
CledaraBot
observe
signed
Clickagy Intelligence Bot
ai assistant
Clickagy
observe
Clickagy Intelligence Bot v2
ai assistant
Clickagy
observe
ClickUpLinkUnfurler
ai assistant
ClickUp
ClickUpLinkUnfurler
observe
Cliqzbot
other
Cliqzbot
observe
Cloudflare AutoRAG
ai crawler
Cloudflare
Cloudflare-AutoRAG
observe
Cloudflare Browser Run
ai assistant
Cloudflare
cloudflare-browser-rendering
observe
signed
Cloudflare Crawler
ai crawler
Cloudflare
Cloudflare Custom Hostname Verification
observe
signed
Cloudflare CSUP
other
Cloudflare
cloudflare-csup
observe
Cloudflare Validator
other
Cloudflare
Cloudflare-Validator
observe
CloudFlare-AlwaysOnline
other
CloudFlare-AlwaysOnline
observe
cloudflare-csup
other
cloudflare-csup
observe
CloudflareDiagnostics
other
CloudflareDiagnostics
observe