Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,426 results
SBIntuitionsBot
ai scraper
SB Intuitions
SBIntuitionsBot
observe
SBL-BOT
other
SBL-BOT
observe
ScaleUpXQ-Crawler
other
ScaleUpXQ-Crawler
observe
ScalinthBot
ai assistant
Scalinth
ScalinthBot
observe
ScannerBot
other
ScannerBot
observe
ScavoBot
ai assistant
Scavo
ScavoBot
observe
signed
Schema-Markup-Validator
other
Schema-Markup-Validator
observe
scoop.it
ai assistant
Meltwater
scoop.it
observe
scoopit-crawler
ai assistant
Scoop.it
scoopit-crawler
observe
Scope3
ai assistant
Scope3
Scope3
observe
score3
other
score3
observe
scour.ing
ai assistant
Scour
scour.ing
observe
ScourRSSBot
ai assistant
Scour
ScourRSSBot
observe
ScoutJet
other
ScoutJet
observe
[email protected]
ai assistant
The New York Times
[email protected]
observe
scribdbot
other
scribdbot
observe
Scrunchbot
ai coding agent
Scrunch
Scrunchbot
observe
ScryBot
ai data provider
Unflatten
ScryBot
observe
SearchmetricsBot
other
SearchmetricsBot
observe
searchpreview
other
searchpreview
observe
Security-Audit
other
Security-Audit
observe
SecurityResearch
other
SecurityResearch
observe
Seekbot
other
Seek Bot
Seekbot
observe
Seekr
other
Seekr
observe
seewithkids
other
seewithkids
observe
SeltzSitemapFetcher
other
SeltzSitemapFetcher
observe
semantic-visions-analytica
ai assistant
Semantic Visions
semantic-visions-analytica
observe
semantic-visions-discovery
other
semantic-visions-discovery
observe
semantic-visions-infoseeker
ai assistant
Semantic Visions
semantic-visions-infoseeker
observe
semantic-visions.com crawler
other
semantic-visions.com
observe
Semanticbot
other
Semanticbot
observe
SemanticVisionsBot
ai assistant
Semantic Visions
SemanticVisionsBot
observe
sempi.tech
other
sempi.tech
observe
SemrushBot-BM
other
SemrushBot-BM
observe
SemrushBot-OCOB
ai crawler
Semrush
SemrushBot-OCOB
observe
SemrushBotSwa
ai crawler
Semrush
SemrushBot-SWA
observe
SenderBot
ai assistant
Sender Online Marketing
SenderBot
observe
SentiBot
ai assistant
SentiOne
SentiBot
observe
SentryUptimeBot
ai coding agent
Sentry
SentryUptimeBot
observe
SeoCherryBot
other
SeoCherryBot
observe
SEOkicks-Robot
other
SEOkicks-Robot
observe
SeoLyt
other
SeoLyt
observe
seoscanners
other
seoscanners
observe
seostar.co
other
seostar.co
observe
SEOstats
other
SEOstats
observe
SequelWP Uptime Monitoring Bot
ai coding agent
Nexigen Digital
observe
serendeputybot
ai assistant
Serendeputy
serendeputybot
observe
Server Density Service Monitoring
ai coding agent
StackPath
observe
ServerHunterSpider
other
Server Hunter
ServerHunterSpider
observe
SFDC-Callout
ai assistant
Salseforce
SFDC-Callout
observe
Shap-User
ai assistant
Parallel
Shap-User
observe
ShapBot
ai crawler
Parallel
ShapBot
observe
ShellBot
other
ShellBot
observe
ShortPixel
ai assistant
ShortPixel
ShortPixel
observe
Sidetrade indexer bot
other
Sidetrade
observe
Silktide
ai coding agent
Silktide
Silktide
observe
Silvestro
ai assistant
Silverstream AI
silvestro
observe
signed
SimpleAsyncPageCrawler
other
SimpleAsyncPageCrawler
observe
simplecrawler
other
simplecrawler
observe
SimplePie
ai assistant
SimplePie
SimplePie
observe