Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,426 results
BubblesBot
ai assistant
Bubbles
BubblesBot
observe
BUbiNG
other
BUbiNG
observe
BublupBot
ai assistant
Bublup
BublupBot
observe
Buck
ai assistant
Hypefactors
Buck
observe
Buddybot
other
Buddybot
observe
Bugsnag script fetcher
ai coding agent
Bugsnag
observe
BuiltBotTough
other
BuiltBotTough
observe
Bullseye
other
Bullseye
observe
BunnySlippers
other
BunnySlippers
observe
Buttondown
ai assistant
Buttondown
Buttondown
observe
buzzbot
other
buzzbot
observe
Bytespider
ai scraper
ByteDance
Bytespider
observe
CandidateCrawler
other
CandidateCrawler
observe
CandidOrgWebCheck
ai assistant
Candid
CandidOrgWebCheck
observe
CapsuleChecker
other
CapsuleChecker
observe
CapterraBot
other
CapterraBot
observe
captify-crawler
other
Captify
captify-crawler
observe
carbon-umbrella-bot
other
carbon-umbrella-bot
observe
careerbot
other
careerbot
observe
CartAI Agentic Commerce
ai agent
CartAI
observe
CartAI Agentic Commerce as a Service
ai assistant
CartAI
cartai-agentic-commerce-as-a-service
observe
signed
CATExplorador
other
CATExplorador
observe
caveman-hunter
other
caveman-hunter
observe
CC Metadata Scaper
other
observe
CCBot
ai scraper
Common Crawl
CCBot
observe
rDNS
CDSCbot
other
CDSCbot
observe
Cegbfeieh
other
Cegbfeieh
observe
CensysInspect
security
Censys
CensysInspect
observe
CentComBot
other
MelonNET
CentComBot
observe
Centro Ads.txt Crawler
other
Centro
observe
centryx-domain-mill
ai assistant
Centryx
centryx-domain-mill
observe
Centurybot
other
Centurybot
observe
changedetection
other
changedetection
observe
ChatGLM-Spider
ai scraper
Zhipu AI
ChatGLM-Spider
observe
ChatGPT agent
ai assistant
OpenAI
ChatGPT
observe
signed
ChatGPT-User
ai assistant
OpenAI
ChatGPT-User
observe
CheckMarkNetwork
ai assistant
CheckMark Network
CheckMarkNetwork
observe
CheeseBot
other
CheeseBot
observe
CherryPicker
other
CherryPicker
observe
CherryPickerElite
other
CherryPickerElite
observe
CherryPickerSE
other
CherryPickerSE
observe
ChimeUnfurlAgent
other
ChimeUnfurlAgent
observe
Chirp
ai coding agent
Binary Canary
Chirp
observe
Cincraw
ai assistant
CINC
Cincraw
observe
cis5550-crawler
other
cis5550-crawler
observe
Cision
other
Cision
observe
CISPA Webcrawler
other
CISPA
observe
citeseerxbot
other
citeseerxbot
observe
CitibotSiteCrawler
ai crawler
Citibot
CitibotSiteCrawler
observe
Citoid
ai assistant
Wikimedia
Citoid
observe
Claude
ai crawler
Anthropic
ClaudeBot
observe
Claude-Code
ai coding agent
Anthropic
Claude-Code
observe
Claude-SearchBot
ai crawler
Anthropic
Claude-SearchBot
observe
Claude-User
ai assistant
Anthropic
Claude-User
observe
Claude-Web
ai agent
Anthropic
Claude-Web
observe
ClaudeBot
ai scraper
Anthropic
ClaudeBot
observe
Cledara SaaS Management Agent
ai assistant
Cledara
CledaraBot
observe
signed
CledaraBot
ai agent
Cledara
observe
Clickagy Intelligence Bot
ai assistant
Clickagy
observe
Clickagy Intelligence Bot v2
ai assistant
Clickagy
observe