Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,426 results
UptimeStatistics
ai coding agent
UptimeStatistics
UptimeStatistics
observe
URL Control
other
observe
URL_Spider_Pro
other
URL_Spider_Pro
observe
urlappendbot
other
urlappendbot
observe
UrlBeeBot
other
UrlBeeBot
observe
UrlDownloadabilityBot
other
UrlDownloadabilityBot
observe
urlsuma.de crawler
other
urlsuma.de
observe
URLy Warning
other
URLy
observe
usasearch
other
usasearch
observe
UseAI
ai assistant
Use AI
UseAI
observe
UsineNouvelleCrawler
other
UsineNouvelleCrawler
observe
UT-Dorkbot
other
UT-Dorkbot
observe
Validator.nu
ai coding agent
W3C
Validator.nu
observe
VCI
other
VCI
observe
VCI WebViewer VCI WebViewer Win32
other
observe
vebidoobot
other
vebidoobot
observe
vecteurplus
other
vecteurplus
observe
VelenPublicWebCrawler
ai scraper
Velen
VelenPublicWebCrawler
observe
Ventus Agent
ai assistant
ventus-agent
observe
signed
Veoozbot
other
Veoozbot
observe
Verity
ai assistant
GumGum
Verity
observe
verticalsearch
other
verticalsearch
observe
VeryHip
other
VeryHip
observe
VettedAiBot
other
VettedAiBot
observe
Viber
ai assistant
Viber
Viber
observe
Vigil
other
Vigil
observe
Vigilare
other
Vigilare
observe
virustotal
ai assistant
VirusTotal
virustotal
observe
Visually.io Shopify Editor
ai assistant
Visually.io
visually-io-shopify-editor
observe
signed
VKRobot
other
VKRobot
observe
vkShare
ai assistant
VK
vkShare
observe
vmcrawl
other
vmcrawl
observe
VoilaBot
other
VoilaBot
observe
voltron
other
voltron
observe
VoluumDSP-content-bot
other
VoluumDSP-content-bot
observe
vsw
other
vsw
observe
vuhuvBot
other
vuhuvBot
observe
vuln_scanner
other
vuln_scanner
observe
W3C_CSS_Validator
ai coding agent
W3C
W3C_CSS_Validator
observe
W3C_I18n-Checker
other
W3C_I18n-Checker
observe
W3C_Unicorn
other
W3C_Unicorn
observe
W3C_Validator
ai coding agent
W3C
W3C_Validator
observe
W3C-checklink
ai coding agent
W3C
W3C-checklink
observe
W3C-mobileOK
other
W3C-mobileOK
observe
WadooBot
other
WadooBot
observe
WanScannerBot
other
WanScannerBot
observe
WappalyzerBot
ai assistant
Wappalyzer
WappalyzerBot
observe
WARDBot
ai crawler
WEBSPARK
WARDBot
observe
WASALive-Bot
other
WASALive-Bot
observe
WAVE Streaming Platform
ai assistant
wave-streaming-platform
observe
signed
Wayflyer
other
Wayflyer
Wayflyer
observe
wbsearchbot
other
wbsearchbot
observe
Web Image Collector
other
observe
web-archive-net.com.bot
other
web-archive-net.com.bot
observe
WebAuto
other
WebAuto
observe
WebBandit
other
WebBandit
observe
WebCapture 2.0
other
WebCapture
observe
webcompanycrawler
other
webcompanycrawler
observe
WebCopier
other
WebCopier
observe
WebCopier v.2.2
other
observe