Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
290 results · AI Assistant
UpContent
Sottopop-user-rss
UpContent
SottopopNone
SpyglassesBot
Start.me
StartmeBot
Valve Software
SteamChat
Substack
SubstackContentFetch
Superfeedr
Superfeedr
Sur.ly
SurdotlyBot
Matrix
Synapse
Taboola
Taboolabot
TechSpy
TechSpyBot
TheSkinnyCrawler
ByteDance
TikTokSpider
Tiny Tiny RSS
Tiny
Alibaba
TongyiBot
Trendsmap
TrendsmapResolver
Meltwater
TweetmemeBot
Twin
TwinAgent
Twingly
Twingly Recon-Klondike
Twingly
Semasio
uipbot
Ubermetrics
um-FC
Unicepta
um-IC
Unicepta
um-LN
UpContent
UpContent
UpContent
UpContent-QA
UpContent
UpContent-Test
Upstash
Upstash-QStash
Use AI
UseAI
ventus-agent
GumGum
Verity
Viber
Viber
VirusTotal
virustotal
Visually.io
visually-io-shopify-editor
VK
vkShare
wave-streaming-platform
Weborama
weborama-fetcher
AspenTechLabs
webspidermount
Affsignal
WebTrackrCrawler
Webz.io
webzio
Meta
WhatsApp
Wire
WireReaderBot
Wunderkind
wknd-bot
WOVN
WovnCrawler
Yandex
YaDirectFetcher
Linkfluence
YaK
Yelp
YelpBot
Baidu
YiyanBot
Zipchat Inc
ZipchatBot
Wikimedia
ZoteroTranslationServer