Crawler Catalog
1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
74 results · Security
Accessible Web
AccessibleWebBot
Invicti Security
Acunetix
AI Trust Radar
AITrustRadar
Alibaba
AliyunSecBot
Canadian Centre for Child Protection
Black Duck
Black
BlackVeil
BlackVeil-Security-Scanner
Blockaid
Blockaid
Cloudflare
bushbaby
Censys
CensysInspect
Chief Tools
CertChief
Qualys
cf-qualys-scanner
Cloudflare
Cloudflare
CookieHub
Cookie
Cookie Maestro
cookie-maestro
Cybot A/S
Cookiebot
CookieHub
CookieHubVerify
CookieYes
CookieYesbot
CyberOK
CyberOKInspect
Cypex
cypex.ai/scanning
Detectify
Detectify
DigiCert
DigiCert
Drata
Drata
Dratabot
Palo Alto Networks
Expanse
Black Duck
FastDAST
Google
Google-Safety
Google-Site-Verification
Google-Trust-Services
Hardenize
Hardenize
Infrawatch
Infrawatch
Driftnet
InternetMeasurement
ISS
ISSCyberRiskCrawler
Coinbase
Jumio
K7 Computing
K7MLWCBot
Legal Monster ApS
LegalMonster
Let's Encrypt
Netcraft
Netcraft
Open-source
Nmap
Nmap
Observer
Observer
ODIN
Odin
OKX
OneTrust
OneTrust
Probely
Unspam Technologies, Inc
ProjectShield-UrlCheck
Qualys
Qualys
Redactus
Reward Gateway
reward-gateway
Sansec Security Monitor
Sansec
Security Headers
SecurityHeaders
Sentry
Sentry
SiteLock
SiteLock
SiteLock
SiteLockSpider
Smarsh
SmarshBot
Ruhr University Bochum
Software