Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,953 results
suzuran
other
suzuran
observe
sv-watchagent
other
sv-watchagent
observe
Svix Webhooks
other
Svix Inc.
sender-9YMgn
allow
Swiftbot
other
Swiftbot
observe
Swifteq Link Checker
monitoring
Swfiteq Ltd
Swifteq
allow
SwifteqLinkChecker
ai coding agent
Swifteq
SwifteqLinkChecker
observe
SWIMGBot
other
SWIMGBot
observe
Synapse
ai assistant
Matrix
Synapse
observe
SynthesiBot
other
SynthesiBot
observe
Synthesio
other
Synthesio
observe
Synthetic Bot
monitoring
DataDog
Synthetic
allow
synthetic-monitoring-agent
ai coding agent
Grafana Labs
synthetic-monitoring-agent
observe
Sysomos
other
Sysomos
observe
Szukacz
other
Szukacz
observe
t3versionsBot
ai coding agent
T3Versions
t3versionsBot
observe
Taboola
seo crawler
Taboola
Taboolabot
challenge
Taboolabot
ai assistant
Taboola
Taboolabot
observe
tagoobot
other
tagoobot
observe
TalarionSearchBot
ai crawler
Talarion
TalarionSearchBot
observe
Talkwater
other
Talkwater
observe
TangibleeBot
other
TangibleeBot
observe
TaraGroup Intelligent Bot
other
TaraGroup
observe
TaskMemoryBot
other
TaskMemoryBot
observe
TavilyBot
ai crawler
Tavily
TavilyBot
observe
TechSpyBot
ai assistant
TechSpy
TechSpyBot
observe
Telegram Bot
social media
Telegram
TelegramBot
allow
Teleport
other
Teleport
observe
TeleportPro
other
TeleportPro
observe
Telesoft
other
Telesoft
observe
Teoma
search engine
Ask
Teoma
allow
Termly
other
Termly
TermlyBot
observe
TermlyBot
monitoring
Termly
TermlyBot
allow
Terra Cotta
ai data provider
Ceramic
observe
Terracotta
search engine
Ceramic
TerraCotta
allow
Testomatobot
ai coding agent
Testomato
Testomatobot
observe
The Intraformant
other
observe
TheNomad
other
TheNomad
observe
theoldreader.com
other
theoldreader.com
observe
TheSkinnyAI
ai assistant
TheSkinnyCrawler
observe
signed
Thinkbot
other
Thinkbot
observe
thinkers-bot
other
thinkers-bot
observe
Thinklab
other
Thinklab
observe
Thousand Eyes Cloud Agent
monitoring
Thousand Eyes
Thousand
allow
TigerBot
other
TigerBot
observe
TikTokSpider
ai assistant
ByteDance
TikTokSpider
observe
Timpibot
ai scraper
Timpi
Timpibot
observe
TinEye
search engine
TinEye
TinEye
allow
TinEye-bot
search engine
TinEye
TinEye-bot
allow
TinEye-bot-live
search engine
TinEye
TinEye-bot-live
allow
TinEye-Web
search engine
TinEye
TinEye-Web
allow
Tiny
ai assistant
Tiny Tiny RSS
Tiny
observe
Titan
other
Titan
observe
TLDWPBot
other
TLDWPBot
observe
TLS tester
other
observe
toCrawl
other
toCrawl
observe
TombaPublicWebCrawler
other
TombaPublicWebCrawler
observe
TongyiBot
ai assistant
Alibaba
TongyiBot
observe
TopifyBot
other
TopifyBot
observe
Toplistbot
other
Toplistbot
observe
TopStoriesBot
other
TopStoriesBot
observe