Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
844 results · Other
SubspaceBot
other
SubspaceBot
observe
Sucuri Integrity Monitor
other
observe
Sucuri Uptime Monitor
other
observe
SumbleBot
other
SumbleBot
observe
SummalyBot
other
SummalyBot
observe
Summerflare
other
Summerflare
observe
Summify
other
Summify
observe
Supabase Paired Crawler
other
Supabase
observe
SuperBot
other
SuperBot
observe
superpagesbot2.0
other
superpagesbot2.0
observe
SurliBot
other
SurliBot
observe
SurveyBot
other
SurveyBot
observe
suzuran
other
suzuran
observe
sv-watchagent
other
sv-watchagent
observe
Svix Webhooks
other
Svix Inc.
sender-9YMgn
allow
Swiftbot
other
Swiftbot
observe
SWIMGBot
other
SWIMGBot
observe
SynthesiBot
other
SynthesiBot
observe
Synthesio
other
Synthesio
observe
Sysomos
other
Sysomos
observe
Szukacz
other
Szukacz
observe
tagoobot
other
tagoobot
observe
Talkwater
other
Talkwater
observe
TangibleeBot
other
TangibleeBot
observe
TaraGroup Intelligent Bot
other
TaraGroup
observe
TaskMemoryBot
other
TaskMemoryBot
observe
Teleport
other
Teleport
observe
TeleportPro
other
TeleportPro
observe
Telesoft
other
Telesoft
observe
Termly
other
Termly
TermlyBot
observe
The Intraformant
other
observe
TheNomad
other
TheNomad
observe
theoldreader.com
other
theoldreader.com
observe
Thinkbot
other
Thinkbot
observe
thinkers-bot
other
thinkers-bot
observe
Thinklab
other
Thinklab
observe
TigerBot
other
TigerBot
observe
Titan
other
Titan
observe
TLDWPBot
other
TLDWPBot
observe
TLS tester
other
observe
toCrawl
other
toCrawl
observe
TombaPublicWebCrawler
other
TombaPublicWebCrawler
observe
TopifyBot
other
TopifyBot
observe
Toplistbot
other
Toplistbot
observe
TopStoriesBot
other
TopStoriesBot
observe
ToutiaoSpider
other
ToutiaoSpider
observe
Traackr.com
other
Traackr.com
observe
tracemyfile
other
tracemyfile
observe
trafilatura
other
trafilatura
observe
Trapezohedron
other
Trapezohedron
observe
trendeo
other
trendeo
observe
trendkite-akashic-crawler
other
trendkite-akashic-crawler
observe
trendybuzz
other
trendybuzz
observe
trovitBot
other
trovitBot
observe
True_Robot
other
True_Robot
observe
TruliaBot
other
TruliaBot
observe
Trustly
other
Trustly Group AB
allow
TSMbot
other
TSMbot
observe
turingos
other
turingos
observe
TweetedTimes
other
TweetedTimes
observe