Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
81 results · Security
Sansec Security Monitor
security
Sansec Security Monitor
Sansec
allow
SecurityHeaders
security
Security Headers
SecurityHeaders
allow
Sentry
security
Sentry
https://sentry.io
allow
SiteLock
security
SiteLock
SiteLock (AccMan Connector)
allow
SiteLockSpider
security
SiteLock
SiteLockSpider
allow
SmarshBot
security
Smarsh
SmarshBot
allow
Software Security Research
security
Ruhr University Bochum
Software
allow
Splunk Attack Analyzer
security
Splunk
TwinWaveScanner
allow
sqlmap
security
Open-source
challenge
SSL Labs
security
Qualys
allow
Sucuri
security
Sucuri
Sucuri
allow
TwinWaveScanner
security
Splunk
TwinWaveScanner
allow
Usercentricsbot
security
Usercentrics
Usercentricsbot
allow
Verispider
security
Project Honey Pot
Verispider
allow
VirusTotalScan
security
VirusTotal
VirusTotalScan
allow
visionheight.com/scan
security
visionheight.com/scan
allow
WebTotemBot
security
WebTotem
WTotem
allow
Wordfence Central
security
Wordfence
allow
WPScan
security
WPScan LLC
WPScan
challenge
WTotem
security
WebTotem
WTotem
allow
WTotemBot
security
WebTotem
WTotemBot
allow