Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
2,114 results
UbiCrawler
other
UbiCrawler
observe
UGAResearchAgent
other
UGAResearchAgent
observe
uipbot
ai assistant
Semasio
uipbot
observe
um-FC
ai assistant
Ubermetrics
um-FC
observe
um-IC
ai assistant
Unicepta
um-IC
observe
um-LN
ai assistant
Unicepta
um-LN
observe
UMichSearchEngineCrawler
other
UMichSearchEngineCrawler
observe
UpContent
ai assistant
UpContent
UpContent
observe
UpContent-QA
ai assistant
UpContent
UpContent-QA
observe
UpContent-Test
ai assistant
UpContent
UpContent-Test
observe
upday
seo crawler
upday GmbH & Co. KG
upday
challenge
updown.io
ai coding agent
Updown
updown.io
observe
updown.io daemon
ai coding agent
updown.io
observe
UpDownBot
monitoring
Updown
UpDownBot
allow
Updownerbot
other
Updownerbot
observe
Upflow
other
Upflow
observe
Upstash QStash
ai assistant
Upstash
Upstash-QStash
observe
signed
Uptime LLC
monitoring
Uptime
Uptime
allow
Uptime Monitoring
monitoring
GoDaddy
UptimeMonitoring
allow
Uptime-Kuma
other
Uptime-Kuma
observe
UptimeBot
monitoring
UptimeBot
UptimeBot
allow
Uptimebot.org
other
Uptimebot.org
observe
uptimedoctor
ai coding agent
uptimedoctor
uptimedoctor
observe
UptimeRobot
monitoring
Uptimerobot
UptimeRobot
allow
rDNS
UptimeStatistics
ai coding agent
UptimeStatistics
UptimeStatistics
observe
Uptimia
monitoring
Uptimia
Uptimia
allow
UptrendsBot
monitoring
Uptrends GmbH
uptrends
allow
URL Control
other
observe
URL_Spider_Pro
other
URL_Spider_Pro
observe
urlappendbot
other
urlappendbot
observe
UrlBeeBot
other
UrlBeeBot
observe
UrlDownloadabilityBot
other
UrlDownloadabilityBot
observe
urlsuma.de crawler
other
urlsuma.de
observe
URLy Warning
other
URLy
observe
usasearch
other
usasearch
observe
UseAI
ai assistant
Use AI
UseAI
observe
Usercentricsbot
security
Usercentrics
Usercentricsbot
allow
UsineNouvelleCrawler
other
UsineNouvelleCrawler
observe
UT-Dorkbot
other
UT-Dorkbot
observe
Validator.nu
ai coding agent
W3C
Validator.nu
observe
VaultPress
other
Automattic
VaultPress
allow
VCI
other
VCI
observe
VCI WebViewer VCI WebViewer Win32
other
observe
vebidoobot
other
vebidoobot
observe
vecteurplus
other
vecteurplus
observe
VelenPublicWebCrawler
ai scraper
Velen
VelenPublicWebCrawler
observe
Ventus Agent
ai assistant
ventus-agent
observe
signed
Veoozbot
other
Veoozbot
observe
Verispider
security
Project Honey Pot
Verispider
allow
Verity
ai assistant
GumGum
Verity
observe
verticalsearch
other
verticalsearch
observe
VeryHip
other
VeryHip
observe
VettedAiBot
other
VettedAiBot
observe
Viber
ai assistant
Viber
Viber
observe
videootv Bot
seo crawler
Digital Green
videootvBot
challenge
Vigil
other
Vigil
observe
Vigilare
other
Vigilare
observe
virustotal
ai assistant
VirusTotal
virustotal
observe
VirusTotalScan
security
VirusTotal
VirusTotalScan
allow
visionheight.com/scan
security
visionheight.com/scan
allow