Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
2,114 results
Streamline3Bot
other
Streamline3Bot
observe
Stripe
other
Stripe
allow
stripe_apple_pay_verify_ownership
other
Stripe
allow
Stripebot
other
Stripe
Stripebot
allow
SubspaceBot
other
SubspaceBot
observe
SubstackContentFetch
ai assistant
Substack
SubstackContentFetch
observe
Sucuri
security
Sucuri
Sucuri
allow
Sucuri Integrity Monitor
other
observe
Sucuri Uptime Monitor
other
observe
SumbleBot
other
SumbleBot
observe
SummalyBot
other
SummalyBot
observe
Summerflare
other
Summerflare
observe
Summify
other
Summify
observe
Supabase Paired Crawler
other
Supabase
observe
SuperBot
other
SuperBot
observe
Superfeedr
ai assistant
Superfeedr
Superfeedr
observe
superpagesbot2.0
other
superpagesbot2.0
observe
SurdotlyBot
ai assistant
Sur.ly
SurdotlyBot
observe
SurliBot
other
SurliBot
observe
SurveyBot
other
SurveyBot
observe
suzuran
other
suzuran
observe
sv-smarttvscan
other
sv-smarttvscan
observe
sv-watchagent
other
sv-watchagent
observe
Svix Webhooks
other
Svix Inc.
sender-9YMgn
allow
Swiftbot
other
Swiftbot
observe
Swifteq Link Checker
monitoring
Swfiteq Ltd
Swifteq
allow
SwifteqLinkChecker
ai coding agent
Swifteq
SwifteqLinkChecker
observe
SWIMGBot
other
SWIMGBot
observe
Synapse
ai assistant
Matrix
Synapse
observe
SynthesiBot
other
SynthesiBot
observe
Synthesio
other
Synthesio
observe
Synthetic Bot
monitoring
DataDog
Synthetic
allow
synthetic-monitoring-agent
ai coding agent
Grafana Labs
synthetic-monitoring-agent
observe
Sysomos
other
Sysomos
observe
Szukacz
other
Szukacz
observe
t3versionsBot
ai coding agent
T3Versions
t3versionsBot
observe
Taboola
seo crawler
Taboola
Taboolabot
challenge
Taboolabot
ai assistant
Taboola
Taboolabot
observe
tagoobot
other
tagoobot
observe
Taktikal-Scanner
other
Taktikal
Taktikal-Scanner
observe
TalarionFrontierCrawler
ai data provider
Talarion
TalarionFrontierCrawler
observe
TalarionSearchBot
ai crawler
Talarion
TalarionSearchBot
observe
Talkwater
other
Talkwater
observe
TangibleeBot
other
TangibleeBot
observe
TaraGroup Intelligent Bot
other
TaraGroup
observe
TaskMemoryBot
other
TaskMemoryBot
observe
TavilyBot
ai crawler
Tavily
TavilyBot
observe
TechSpyBot
ai assistant
TechSpy
TechSpyBot
observe
Telegram Bot
social media
Telegram
TelegramBot
allow
Teleport
other
Teleport
observe
TeleportPro
other
TeleportPro
observe
Telesoft
other
Telesoft
observe
Teoma
search engine
Ask
Teoma
allow
Termly
other
Termly
TermlyBot
observe
TermlyBot
monitoring
Termly
TermlyBot
allow
Terracotta
search engine
Ceramic
TerraCotta
allow
Testomatobot
ai coding agent
Testomato
Testomatobot
observe
TextRazor Downloader
ai assistant
TextRazor
observe
The Intraformant
other
observe
TheNomad
other
TheNomad
observe