Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,953 results
Bravebot
ai crawler
Brave Software, Inc.
Bravebot
observe
Brightbot
ai data provider
Bright Data
Brightbot
observe
BrightEdge Bot
seo crawler
BrightEdge
BrightEdge
challenge
brokenlinkcheck.com
other
brokenlinkcheck.com
observe
BrokenLinksBot
other
BrokenLinksBot
observe
Browserbase
ai assistant
Browserbase
Browserbase
observe
signed
Brynson Compliance Bot
monitoring
Brynson
BrynsonComplianceBot
allow
signed
BTWebClient
other
BTWebClient
observe
BubblesBot
ai assistant
Bubbles
BubblesBot
observe
BUbiNG
other
BUbiNG
observe
BublupBot
ai assistant
Bublup
BublupBot
observe
Buck
ai assistant
Hypefactors
Buck
observe
Buddybot
other
Buddybot
observe
Buffer Link Preview Bot
social media
Buffer
BufferLinkPreviewBot
allow
BugsNag
monitoring
Bugs Nag
BugsNag
allow
Bugsnag script fetcher
ai coding agent
Bugsnag
observe
BuiltBotTough
other
BuiltBotTough
observe
BuiltWith
monitoring
BuiltWith
BuiltWith
allow
Bullseye
other
Bullseye
observe
BunnySlippers
other
BunnySlippers
observe
Bushbaby
security
Cloudflare
bushbaby
allow
Buttondown
ai assistant
Buttondown
Buttondown
observe
Buttondown RSS-Feed-Parser
social media
Buttondown
Buttondown RSS-Feed-Parser
allow
buzzbot
other
buzzbot
observe
Bytespider
ai scraper
ByteDance
Bytespider
observe
CaliberBot
seo crawler
Conductor
Caliperbot
challenge
CandidateCrawler
other
CandidateCrawler
observe
Capital One Bot
seo crawler
Capital One
CapitalOneBot
challenge
CapsuleChecker
other
CapsuleChecker
observe
CapterraBot
other
CapterraBot
observe
captify-crawler
other
Captify
captify-crawler
observe
carbon-umbrella-bot
other
carbon-umbrella-bot
observe
careerbot
other
careerbot
observe
CartAI Agentic Commerce as a Service
ai assistant
CartAI
cartai-agentic-commerce-as-a-service
observe
signed
catchpoint
monitoring
Catchpoint
Catchpoint
allow
caveman-hunter
other
caveman-hunter
observe
CC Metadata Scaper
other
observe
cc_bot
other
cc_bot
observe
CCBot
ai scraper
Common Crawl
CCBot
observe
rDNS
CDSCbot
other
CDSCbot
observe
Cegbfeieh
other
Cegbfeieh
observe
CensysInspect
security
Censys
CensysInspect
observe
CentComBot
other
MelonNET
CentComBot
observe
Centro Ads.txt Crawler
other
Centro
observe
Centurybot
other
Centurybot
observe
Cert Chief
security
Chief Tools
CertChief
allow
CertChief
security
Chief
CertChief
allow
cf-qualys-scanner
security
Qualys
cf-qualys-scanner
allow
changedetection
other
changedetection
observe
Channable
seo crawler
Channable
channable/importer
challenge
channable
ai assistant
Channable
channable
observe
Channel3Bot
seo crawler
Channel3
Channel3Bot
challenge
ChargeBee
other
ChargeBee
ChargeBee
allow
ChatGLM-Spider
ai scraper
Zhipu AI
ChatGLM-Spider
observe
ChatGPT agent
ai assistant
OpenAI
ChatGPT
observe
signed
ChatGPT-User
ai assistant
OpenAI
ChatGPT-User
observe
Checkly Bot
monitoring
Checkly
Checkly
allow
CheckMarkNetwork
ai assistant
CheckMark Network
CheckMarkNetwork
observe
CheckView
monitoring
CheckView.io
CheckView
allow
CheeseBot
other
CheeseBot
observe