Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
81 results · Security
Accessible Web
AccessibleWebBot
Invicti Security
Acunetix
AI Trust Radar
AITrustRadar
Alibaba
AliyunSecBot
Canadian Centre for Child Protection
Black Duck
Black
BlackVeil
BlackVeil-Security-Scanner
Blockaid
Blockaid
Cloudflare
bushbaby
Censys
CensysInspect
Chief Tools
CertChief
Chief
CertChief
Qualys
cf-qualys-scanner
Cloudflare
cloudflare-radar-url-scanner
CookieHub
Cookie
Cookie Maestro
cookie-maestro
Cookie Maestro
cookie-maestro
Cybot A/S
Cookiebot
CookieHub
CookieHub
CookieHub
CookieHubVerify
CookieYes
CookieYesbot
CybaaBot
CyberOK
CyberOKInspect
Cypex
cypex.ai/scanning
Detectify
Detectify
Drata
Dratabot
Drata
Dratabot
Palo Alto Networks
Expanse
Black Duck
FastDAST
Foregenix
Foregenix
Google Inspection Tool
Google-Safety
Google-Site-Verification
Google-Trust-Services
Google-BusinessLinkVerifi
Google-InspectionTool
Hardenize
Hardenize
Infrawatch
Infrawatch
Driftnet
InternetMeasurement
ISS
ISSCyberRiskCrawler
Coinbase
Jumio
K7 Computing
K7MLWCBot
Legal Monster ApS
LegalMonster
Let's Encrypt
Let's Encrypt
Netcraft
Netcraft
Netcraft
Open-source
Nmap
Nmap
Observer
Observer
ODIN
Odin
OKX
OKX-dolphin-crawler
OneTrust
OneTrust
Probely
ProbelySPDR
Unspam Technologies, Inc
Verispider
ProjectShield-UrlCheck
ProjectShield-UrlCheck
Qualys
cf-qualys-scanner
Redactus
RedactusWebClassifier
Reward Gateway
reward-gateway