Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
844 results · Other
twengabot
other
twengabot
observe
Twilio Proxy
other
Twilio, Inc.
allow
Twurly
other
Twurly
observe
UbiCrawler
other
UbiCrawler
observe
UGAResearchAgent
other
UGAResearchAgent
observe
UMichSearchEngineCrawler
other
UMichSearchEngineCrawler
observe
Updownerbot
other
Updownerbot
observe
Upflow
other
Upflow
observe
Uptime-Kuma
other
Uptime-Kuma
observe
Uptimebot.org
other
Uptimebot.org
observe
URL Control
other
observe
URL_Spider_Pro
other
URL_Spider_Pro
observe
urlappendbot
other
urlappendbot
observe
UrlBeeBot
other
UrlBeeBot
observe
urlsuma.de crawler
other
urlsuma.de
observe
URLy Warning
other
URLy
observe
usasearch
other
usasearch
observe
UsineNouvelleCrawler
other
UsineNouvelleCrawler
observe
UT-Dorkbot
other
UT-Dorkbot
observe
VaultPress
other
Automattic
VaultPress
allow
VCI
other
VCI
observe
VCI WebViewer VCI WebViewer Win32
other
observe
vebidoobot
other
vebidoobot
observe
vecteurplus
other
vecteurplus
observe
Veoozbot
other
Veoozbot
observe
verticalsearch
other
verticalsearch
observe
Vigil
other
Vigil
observe
VKRobot
other
VKRobot
observe
vmcrawl
other
vmcrawl
observe
VoilaBot
other
VoilaBot
observe
voltron
other
voltron
observe
VoluumDSP-content-bot
other
VoluumDSP-content-bot
observe
vsw
other
vsw
observe
vuhuvBot
other
vuhuvBot
observe
W3C_I18n-Checker
other
W3C_I18n-Checker
observe
W3C_Unicorn
other
W3C_Unicorn
observe
W3C-mobileOK
other
W3C-mobileOK
observe
WadooBot
other
WadooBot
observe
WanScannerBot
other
WanScannerBot
observe
WASALive-Bot
other
WASALive-Bot
observe
Wayflyer
other
Wayflyer
Wayflyer
observe
wbsearchbot
other
wbsearchbot
observe
Web Image Collector
other
observe
web-archive-net.com.bot
other
web-archive-net.com.bot
observe
WebAuto
other
WebAuto
observe
WebBandit
other
WebBandit
observe
WebCapture 2.0
other
WebCapture
observe
webcompanycrawler
other
webcompanycrawler
observe
WebCopier
other
WebCopier
observe
WebCopier v.2.2
other
observe
WebCopier v3.2a
other
observe
WebCrawler
other
WebCrawler
observe
WebDataStats
other
WebDataStats
observe
WebEnhancer
other
WebEnhancer
observe
WebmasterWorldForumBot
other
WebmasterWorldForumBot
observe
webmon
other
webmon
observe
WebReaper
other
WebReaper
observe
WebSauger
other
WebSauger
observe
WebScraperBot
other
WebScraperBot
observe
websearch-bot
other
websearch-bot
observe