Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,953 results
URL Control
other
observe
URL_Spider_Pro
other
URL_Spider_Pro
observe
urlappendbot
other
urlappendbot
observe
UrlBeeBot
other
UrlBeeBot
observe
urlsuma.de crawler
other
urlsuma.de
observe
URLy Warning
other
URLy
observe
usasearch
other
usasearch
observe
UseAI
ai assistant
Use AI
UseAI
observe
Usercentricsbot
security
Usercentrics
Usercentricsbot
allow
UsineNouvelleCrawler
other
UsineNouvelleCrawler
observe
UT-Dorkbot
other
UT-Dorkbot
observe
Validator.nu
ai coding agent
W3C
Validator.nu
observe
VaultPress
other
Automattic
VaultPress
allow
VCI
other
VCI
observe
VCI WebViewer VCI WebViewer Win32
other
observe
vebidoobot
other
vebidoobot
observe
vecteurplus
other
vecteurplus
observe
VelenPublicWebCrawler
ai scraper
Velen
VelenPublicWebCrawler
observe
Ventus Agent
ai assistant
ventus-agent
observe
signed
Veoozbot
other
Veoozbot
observe
Verispider
security
Project Honey Pot
Verispider
allow
Verity
ai assistant
GumGum
Verity
observe
verticalsearch
other
verticalsearch
observe
Viber
ai assistant
Viber
Viber
observe
videootv Bot
seo crawler
Digital Green
videootvBot
challenge
Vigil
other
Vigil
observe
virustotal
ai assistant
VirusTotal
virustotal
observe
VirusTotalScan
security
VirusTotal
VirusTotalScan
allow
visionheight.com/scan
security
visionheight.com/scan
allow
Visually.io Shopify Editor
ai assistant
Visually.io
visually-io-shopify-editor
observe
signed
VKRobot
other
VKRobot
observe
vkShare
ai assistant
VK
vkShare
observe
vmcrawl
other
vmcrawl
observe
VoilaBot
other
VoilaBot
observe
voltron
other
voltron
observe
VoluumDSP-content-bot
other
VoluumDSP-content-bot
observe
vsw
other
vsw
observe
vuhuvBot
other
vuhuvBot
observe
W3 Validator Services
seo crawler
World Wide Web Consortium (W3C)
W3
challenge
W3C_CSS_Validator
ai coding agent
W3C
W3C_CSS_Validator
observe
W3C_I18n-Checker
other
W3C_I18n-Checker
observe
W3C_Unicorn
other
W3C_Unicorn
observe
W3C_Validator
ai coding agent
W3C
W3C_Validator
observe
W3C-checklink
ai coding agent
W3C
W3C-checklink
observe
W3C-mobileOK
other
W3C-mobileOK
observe
WadooBot
other
WadooBot
observe
WanScannerBot
other
WanScannerBot
observe
WARDBot
ai crawler
WEBSPARK
WARDBot
observe
WASALive-Bot
other
WASALive-Bot
observe
Watchbot
monitoring
WatchBot
Watchbot
allow
Watchful
monitoring
Watchful LLC
Watchful
allow
WAVE Streaming Platform
ai assistant
wave-streaming-platform
observe
signed
Wayflyer
other
Wayflyer
Wayflyer
observe
wbsearchbot
other
wbsearchbot
observe
Web Image Collector
other
observe
web-archive-net.com.bot
other
web-archive-net.com.bot
observe
WebAuto
other
WebAuto
observe
WebBandit
other
WebBandit
observe
WebCapture 2.0
other
WebCapture
observe
webcompanycrawler
other
webcompanycrawler
observe