Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,284 results
SQWatcher
other
SQWatcher
observe
Stape
ai coding agent
Stape
Stape
observe
star-finder.de Bot
other
star-finder.de
observe
StartmeBot
ai assistant
Start.me
StartmeBot
observe
StatistikAustria
other
Statistik Bot
StatistikAustria
observe
SteamChat
ai assistant
Valve Software
SteamChat
observe
StorygizeBot
other
StorygizeBot
observe
Streamline3Bot
other
Streamline3Bot
observe
SubspaceBot
other
SubspaceBot
observe
SubstackContentFetch
ai assistant
Substack
SubstackContentFetch
observe
Sucuri Integrity Monitor
other
observe
Sucuri Uptime Monitor
other
observe
SumbleBot
other
SumbleBot
observe
SummalyBot
other
SummalyBot
observe
Summerflare
other
Summerflare
observe
Summify
other
Summify
observe
Supabase Paired Crawler
other
Supabase
observe
SuperBot
other
SuperBot
observe
Superfeedr
ai assistant
Superfeedr
Superfeedr
observe
superpagesbot2.0
other
superpagesbot2.0
observe
SurdotlyBot
ai assistant
Sur.ly
SurdotlyBot
observe
SurliBot
other
SurliBot
observe
SurveyBot
other
SurveyBot
observe
suzuran
other
suzuran
observe
sv-watchagent
other
sv-watchagent
observe
Swiftbot
other
Swiftbot
observe
SwifteqLinkChecker
ai coding agent
Swifteq
SwifteqLinkChecker
observe
SWIMGBot
other
SWIMGBot
observe
Synapse
ai assistant
Matrix
Synapse
observe
SynthesiBot
other
SynthesiBot
observe
Synthesio
other
Synthesio
observe
synthetic-monitoring-agent
ai coding agent
Grafana Labs
synthetic-monitoring-agent
observe
Sysomos
other
Sysomos
observe
Szukacz
other
Szukacz
observe
t3versionsBot
ai coding agent
T3Versions
t3versionsBot
observe
Taboolabot
ai assistant
Taboola
Taboolabot
observe
tagoobot
other
tagoobot
observe
TalarionSearchBot
ai crawler
Talarion
TalarionSearchBot
observe
Talkwater
other
Talkwater
observe
TangibleeBot
other
TangibleeBot
observe
TaraGroup Intelligent Bot
other
TaraGroup
observe
TaskMemoryBot
other
TaskMemoryBot
observe
TavilyBot
ai crawler
Tavily
TavilyBot
observe
TechSpyBot
ai assistant
TechSpy
TechSpyBot
observe
Teleport
other
Teleport
observe
TeleportPro
other
TeleportPro
observe
Telesoft
other
Telesoft
observe
Termly
other
Termly
TermlyBot
observe
Terra Cotta
ai data provider
Ceramic
observe
Testomatobot
ai coding agent
Testomato
Testomatobot
observe
The Intraformant
other
observe
TheNomad
other
TheNomad
observe
theoldreader.com
other
theoldreader.com
observe
TheSkinnyAI
ai assistant
TheSkinnyCrawler
observe
signed
Thinkbot
other
Thinkbot
observe
thinkers-bot
other
thinkers-bot
observe
Thinklab
other
Thinklab
observe
TigerBot
other
TigerBot
observe
TikTokSpider
ai assistant
ByteDance
TikTokSpider
observe
Timpibot
ai scraper
Timpi
Timpibot
observe