Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
2,114 results
Visually.io Shopify Editor
ai assistant
Visually.io
visually-io-shopify-editor
observe
signed
VivianCrawler
search engine
Evil Source Platform
VivianCrawler
allow
VKRobot
other
VKRobot
observe
vkShare
ai assistant
VK
vkShare
observe
vmcrawl
other
vmcrawl
observe
VoilaBot
other
VoilaBot
observe
voltron
other
voltron
observe
VoluumDSP-content-bot
other
VoluumDSP-content-bot
observe
vsw
other
vsw
observe
vuhuvBot
other
vuhuvBot
observe
vuln_scanner
other
vuln_scanner
observe
W3 Validator Services
seo crawler
World Wide Web Consortium (W3C)
W3
challenge
W3C_CSS_Validator
ai coding agent
W3C
W3C_CSS_Validator
observe
W3C_I18n-Checker
other
W3C_I18n-Checker
observe
W3C_Unicorn
other
W3C_Unicorn
observe
W3C_Validator
ai coding agent
W3C
W3C_Validator
observe
W3C-checklink
ai coding agent
W3C
W3C-checklink
observe
W3C-mobileOK
other
W3C-mobileOK
observe
WadooBot
other
WadooBot
observe
WanScannerBot
other
WanScannerBot
observe
WappalyzerBot
ai assistant
Wappalyzer
WappalyzerBot
observe
WARDBot
ai crawler
WEBSPARK
WARDBot
observe
WASALive-Bot
other
WASALive-Bot
observe
Watchbot
monitoring
WatchBot
Watchbot
allow
Watchful
monitoring
Watchful LLC
Watchful
allow
WAVE Streaming Platform
ai assistant
wave-streaming-platform
observe
signed
Wayflyer
other
Wayflyer
Wayflyer
observe
wbsearchbot
other
wbsearchbot
observe
Web Image Collector
other
observe
web-archive-net.com.bot
other
web-archive-net.com.bot
observe
web-image-collection-research
seo crawler
web-image-collection-research
challenge
WebAuto
other
WebAuto
observe
WebBandit
other
WebBandit
observe
WebCapture 2.0
other
WebCapture
observe
webcompanycrawler
other
webcompanycrawler
observe
WebCopier
other
WebCopier
observe
WebCopier v.2.2
other
observe
WebCopier v3.2a
other
observe
WebCrawler
other
WebCrawler
observe
WebDataStats
other
WebDataStats
observe
WebEnhancer
other
WebEnhancer
observe
WebmasterWorldForumBot
other
WebmasterWorldForumBot
observe
webmon
other
webmon
observe
weborama-fetcher
ai assistant
Weborama
weborama-fetcher
observe
webpagetest
monitoring
WebPageTest
webpagetest
allow
WebReaper
other
WebReaper
observe
WebSauger
other
WebSauger
observe
WebScraperBot
other
WebScraperBot
observe
websearch-bot
other
websearch-bot
observe
Website Quester
other
Website
observe
Website-info.net-Robot
other
Website-info.net-Robot
observe
WebsiteMetadataRetriever
other
WebsiteMetadataRetriever
observe
WebsiteOps
ai coding agent
Watchful
WebsiteOps
observe
websitepulse checker
ai coding agent
WebSitePulse
websitepulse checker
observe
WebsitePulseBot
monitoring
WebsitePulse
WebsitePulseBot
allow
WebSpiderMount
seo crawler
AspenTechLabs Inc
webspidermount
challenge
Webstatus247
monitoring
WebStatus247
Webstatus247
allow
WebStripper
other
WebStripper
observe
webtext-bot
other
webtext-bot
observe
WebTotemBot
security
WebTotem
WTotem
allow