Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
341 results · AI Assistant
TongyiBot
ai assistant
Alibaba
TongyiBot
observe
topicsearch-crawler
ai assistant
FatGrid
topicsearch-crawler
observe
tphotobot
ai assistant
EstiDraft
tphotobot
observe
TrendsmapResolver
ai assistant
Trendsmap
TrendsmapResolver
observe
TweetmemeBot
ai assistant
Meltwater
TweetmemeBot
observe
TwinAgent
ai assistant
Twin
TwinAgent
observe
Twingly
ai assistant
Twingly
Twingly Recon-Klondike
observe
Twingly Recon-Sjostrom
ai assistant
Twingly
observe
uipbot
ai assistant
Semasio
uipbot
observe
um-FC
ai assistant
Ubermetrics
um-FC
observe
um-IC
ai assistant
Unicepta
um-IC
observe
um-LN
ai assistant
Unicepta
um-LN
observe
UpContent
ai assistant
UpContent
UpContent
observe
UpContent-QA
ai assistant
UpContent
UpContent-QA
observe
UpContent-Test
ai assistant
UpContent
UpContent-Test
observe
Upstash QStash
ai assistant
Upstash
Upstash-QStash
observe
signed
UseAI
ai assistant
Use AI
UseAI
observe
Ventus Agent
ai assistant
ventus-agent
observe
signed
Verity
ai assistant
GumGum
Verity
observe
Viber
ai assistant
Viber
Viber
observe
virustotal
ai assistant
VirusTotal
virustotal
observe
Visually.io Shopify Editor
ai assistant
Visually.io
visually-io-shopify-editor
observe
signed
vkShare
ai assistant
VK
vkShare
observe
WappalyzerBot
ai assistant
Wappalyzer
WappalyzerBot
observe
WAVE Streaming Platform
ai assistant
wave-streaming-platform
observe
signed
weborama-fetcher
ai assistant
Weborama
weborama-fetcher
observe
WebTrackrCrawler
ai assistant
Affsignal
WebTrackrCrawler
observe
webzio
ai assistant
Webz.io
webzio
observe
WhatsApp
ai assistant
Meta
WhatsApp
observe
WhiteCircleTrustBot
ai assistant
White Circle
WhiteCircleTrustBot
observe
WhyNoIPv6Bot
ai assistant
Why No IPv6
WhyNoIPv6Bot
observe
WireReaderBot
ai assistant
Wire
WireReaderBot
observe
wknd-bot
ai assistant
Wunderkind
wknd-bot
observe
WovnCrawler
ai assistant
WOVN
WovnCrawler
observe
WP.com mShots
ai assistant
Automattic
observe
YaDirectFetcher
ai assistant
Yandex
YaDirectFetcher
observe
YaK
ai assistant
Linkfluence
YaK
observe
YelpBot
ai assistant
Yelp
YelpBot
observe
YiyanBot
ai assistant
Baidu
YiyanBot
observe
ZipchatBot
ai assistant
Zipchat Inc
ZipchatBot
observe
signed
ZoteroTranslationServer
ai assistant
Wikimedia
ZoteroTranslationServer
observe