Updated daily
Crawler Catalog
1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
1,876 results
BTWebClient
other
BTWebClient
observe
BubblesBot
ai assistant
Bubbles
BubblesBot
observe
BUbiNG
other
BUbiNG
observe
BublupBot
ai assistant
Bublup
BublupBot
observe
Buck
ai assistant
Hypefactors
Buck
observe
Buddybot
other
Buddybot
observe
Buffer Link Preview Bot
social media
Buffer
BufferLinkPreviewBot
allow
BugsNag
monitoring
Bugs Nag
BugsNag
allow
BuiltBotTough
other
BuiltBotTough
observe
BuiltWith
monitoring
BuiltWith
allow
Bullseye
other
Bullseye
observe
BunnySlippers
other
BunnySlippers
observe
Bushbaby
security
Cloudflare
bushbaby
allow
Buttondown
ai assistant
Buttondown
Buttondown
observe
Buttondown RSS-Feed-Parser
social media
Buttondown
Buttondown
allow
buzzbot
other
buzzbot
observe
Bytespider
ai scraper
ByteDance
Bytespider
observe
CaliberBot
seo crawler
Conductor
Caliperbot
challenge
CandidateCrawler
other
CandidateCrawler
observe
Capital One Bot
seo crawler
Capital One
CapitalOneBot
challenge
CapsuleChecker
other
CapsuleChecker
observe
CapterraBot
other
CapterraBot
observe
captify-crawler
other
captify-crawler
observe
carbon-umbrella-bot
other
carbon-umbrella-bot
observe
careerbot
other
careerbot
observe
CartAI Agentic Commerce as a Service
ai assistant
CartAI
observe
signed
catchpoint
monitoring
Catchpoint
Catchpoint
allow
caveman-hunter
other
caveman-hunter
observe
CC
other
CC
observe
CCBot
ai scraper
Common Crawl
CCBot
observe
CDSCbot
other
CDSCbot
observe
Cegbfeieh
other
Cegbfeieh
observe
CensysInspect
security
Censys
CensysInspect
observe
CentComBot
other
MelonNET
CentComBot
observe
Centro
other
Centro
observe
Centurybot
other
Centurybot
observe
Cert Chief
security
Chief Tools
CertChief
allow
cf-qualys-scanner
security
Qualys
cf-qualys-scanner
allow
changedetection
other
changedetection
observe
Channable
seo crawler
Channable
channable
challenge
Channel3Bot
seo crawler
Channel3
Channel3Bot
challenge
ChargeBee
other
ChargeBee
allow
ChatGLM-Spider
ai scraper
Zhipu AI
ChatGLM-Spider
observe
ChatGPT agent
ai assistant
OpenAI
ChatGPT
observe
signed
ChatGPT-User
ai assistant
OpenAI
ChatGPT-User
observe
Checkly Bot
monitoring
Checkly
Checkly
allow
CheckMarkNetwork
ai assistant
CheckMark Network
CheckMarkNetwork
observe
CheckView
monitoring
CheckView.io
allow
CheeseBot
other
CheeseBot
observe
CherryPicker
other
CherryPicker
observe
CherryPickerElite
other
CherryPickerElite
observe
CherryPickerSE
other
CherryPickerSE
observe
ChimeUnfurlAgent
other
ChimeUnfurlAgent
observe
Chirp
ai coding agent
Binary Canary
Chirp
observe
Chrome-Lighthouse
seo crawler
Chrome-Lighthouse
challenge
Cincraw
ai assistant
CINC
Cincraw
observe
cis5550-crawler
other
cis5550-crawler
observe
Cision
other
Cision
observe
CISPA
other
CISPA
observe
citeseerxbot
other
citeseerxbot
observe