Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
2,114 results
BotALot
other
BotALot
observe
Botify
seo crawler
Botify
botify
challenge
BotRightHere
other
BotRightHere
observe
BoxcarBot
other
BoxcarBot
observe
br-crawler
other
br-crawler
observe
brainobot
other
brainobot
observe
BrandONbot
other
BrandONbot
observe
BrandVerity
ai assistant
BrandVerity
BrandVerity
observe
Brandwatch
ai crawler
Brandwatch
magpie-crawler
observe
Bravebot
ai crawler
Brave Software, Inc.
Bravebot
observe
Brightbot
ai data provider
Bright Data
Brightbot
observe
BrightEdge Bot
seo crawler
BrightEdge
BrightEdge
challenge
brokenlinkcheck.com
other
brokenlinkcheck.com
observe
BrokenLinksBot
other
BrokenLinksBot
observe
Browserbase
ai assistant
Browserbase
Browserbase
observe
signed
Brynson Compliance Bot
monitoring
Brynson
BrynsonComplianceBot
allow
signed
BTWebClient
other
BTWebClient
observe
BubblesBot
ai assistant
Bubbles
BubblesBot
observe
BUbiNG
other
BUbiNG
observe
BublupBot
ai assistant
Bublup
BublupBot
observe
Buck
ai assistant
Hypefactors
Buck
observe
Buddybot
other
Buddybot
observe
Buffer Link Preview Bot
social media
Buffer
BufferLinkPreviewBot
allow
BugsNag
monitoring
Bugs Nag
BugsNag
allow
Bugsnag script fetcher
ai coding agent
Bugsnag
observe
BuiltBotTough
other
BuiltBotTough
observe
BuiltWith
monitoring
BuiltWith
BuiltWith
allow
Bullseye
other
Bullseye
observe
BunnySlippers
other
BunnySlippers
observe
Bushbaby
security
Cloudflare
bushbaby
allow
Buttondown
ai assistant
Buttondown
Buttondown
observe
Buttondown RSS-Feed-Parser
social media
Buttondown
Buttondown RSS-Feed-Parser
allow
buzzbot
other
buzzbot
observe
Bytespider
ai scraper
ByteDance
Bytespider
observe
CaliberBot
seo crawler
Conductor
Caliperbot
challenge
CandidateCrawler
other
CandidateCrawler
observe
CandidOrgWebCheck
ai assistant
Candid
CandidOrgWebCheck
observe
Capital One Bot
seo crawler
Capital One
CapitalOneBot
challenge
CapsuleChecker
other
CapsuleChecker
observe
CapterraBot
other
CapterraBot
observe
captify-crawler
other
Captify
captify-crawler
observe
carbon-umbrella-bot
other
carbon-umbrella-bot
observe
careerbot
other
careerbot
observe
CartAI Agentic Commerce
ai agent
CartAI
observe
CartAI Agentic Commerce as a Service
ai assistant
CartAI
cartai-agentic-commerce-as-a-service
observe
signed
catchpoint
monitoring
Catchpoint
Catchpoint
allow
CATExplorador
other
CATExplorador
observe
caveman-hunter
other
caveman-hunter
observe
CC Metadata Scaper
other
observe
CCBot
ai scraper
Common Crawl
CCBot
observe
rDNS
CDSCbot
other
CDSCbot
observe
Cegbfeieh
other
Cegbfeieh
observe
CensysInspect
security
Censys
CensysInspect
observe
CentComBot
other
MelonNET
CentComBot
observe
Centro Ads.txt Crawler
other
Centro
observe
centryx-domain-mill
ai assistant
Centryx
centryx-domain-mill
observe
Centurybot
other
Centurybot
observe
Cert Chief
security
Chief Tools
CertChief
allow
cf-qualys-scanner
security
Qualys
cf-qualys-scanner
allow
CGMSearchBot
search engine
CGMIMM
CGMSearchBot
allow