Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,953 results
Sosospider
other
Sosospider
observe
Sottopop
ai assistant
UpContent
Sottopop
observe
Sottopop-user-rss
ai assistant
UpContent
Sottopop-user-rss
observe
SottopopNone
ai assistant
UpContent
SottopopNone
observe
SpankBot
other
SpankBot
observe
spanner
other
spanner
observe
Spark Shipping
other
Spark Shipping
SparkShipping
allow
SparkpostBot
other
Bird
SparkPost
allow
Spawning AI
ai crawler
Spawning
Spawning-AI
allow
spbot
other
spbot
observe
special_archiver
archiver
Internet Archive
special_archiver
allow
Spectate
monitoring
Spectate
Spectate
allow
SpeedyIndexBot
other
SpeedyIndexBot
observe
Spider
ai scraper
Spider
Spider
observe
SpiderLing
other
SpiderLing
observe
Spinn3r
other
Spinn3r
observe
Splunk
monitoring
Splunk
Rigor
allow
Splunk Attack Analyzer
security
Splunk
TwinWaveScanner
allow
Splunk Synthetics
monitoring
Splunk
Splunk Synthetics
allow
spotter
other
spotter
observe
SpringserveBot
other
SpringserveBot
observe
SputnikBot
other
SputnikBot
observe
Spyglasses Bot
ai assistant
SpyglassesBot
observe
signed
sqlmap
security
Open-source
challenge
SQWatcher
other
SQWatcher
observe
SSL Labs
security
Qualys
allow
Stape
ai coding agent
Stape
Stape
observe
Stape Scanner
monitoring
Stape Inc
Stape
allow
star-finder.de Bot
other
star-finder.de
observe
StartmeBot
ai assistant
Start.me
StartmeBot
observe
Statabot
seo crawler
StataCorp LLC
Statabot
challenge
StatistikAustria
other
Statistik Bot
StatistikAustria
observe
StatsDroneBot
seo crawler
StatsDrone
StatsDroneBot
challenge
StatusCake
monitoring
StatusCake
statuscake
allow
StatusNestBacklinkSpider
seo crawler
StatusNest
StatusNestBacklinkSpider
challenge
Steam Chat
social media
Valve Software
SteamChat
allow
SteamChat
ai assistant
Valve Software
SteamChat
observe
stepstoneCrawlBot
search engine
StepStone
stepstoneCrawlBot
allow
StorygizeBot
other
StorygizeBot
observe
StractBot
search engine
Stract
StractBot
allow
Streamline3Bot
other
Streamline3Bot
observe
Stripe
other
Stripe
allow
stripe_apple_pay_verify_ownership
other
Stripe
allow
Stripebot
other
Stripe
Stripebot
allow
SubspaceBot
other
SubspaceBot
observe
SubstackContentFetch
ai assistant
Substack
SubstackContentFetch
observe
Sucuri
security
Sucuri
Sucuri
allow
Sucuri Integrity Monitor
other
observe
Sucuri Uptime Monitor
other
observe
SumbleBot
other
SumbleBot
observe
SummalyBot
other
SummalyBot
observe
Summerflare
other
Summerflare
observe
Summify
other
Summify
observe
Supabase Paired Crawler
other
Supabase
observe
SuperBot
other
SuperBot
observe
Superfeedr
ai assistant
Superfeedr
Superfeedr
observe
superpagesbot2.0
other
superpagesbot2.0
observe
SurdotlyBot
ai assistant
Sur.ly
SurdotlyBot
observe
SurliBot
other
SurliBot
observe
SurveyBot
other
SurveyBot
observe