Updated daily
Crawler Catalog
1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
1,876 results
seewithkids
other
seewithkids
observe
semantic-visions-discovery
other
semantic-visions-discovery
observe
semantic-visions.com
other
semantic-visions.com
observe
Semanticbot
other
Semanticbot
observe
SemanticScholarBot
search engine
Ai2
SemanticScholarBot
allow
sempi.tech
other
sempi.tech
observe
Semrushbot
seo crawler
Semrush
SemrushBot
challenge
SemrushBot-BA
seo crawler
Semrush
SemrushBot-BA
challenge
SemrushBot-BM
other
SemrushBot-BM
observe
SemrushBot-CT
seo crawler
Semrush
SemrushBot-CT
challenge
SemrushBot-OCOB
ai crawler
Semrush
SemrushBot-OCOB
observe
SemrushBot-SA
seo crawler
Semrush
SemrushBot-SA
challenge
SemrushBotBacklinkAudit
seo crawler
Semrush
challenge
SemrushBotBacklinks
seo crawler
Semrush
SemrushBotBacklinks
challenge
SemrushBotLinkBuilding
seo crawler
Semrush
challenge
SemrushBotSI
seo crawler
Semrush
SemrushBot-SI
challenge
SemrushBotSwa
ai crawler
Semrush
SemrushBot-SWA
observe
SendGrid
other
SendGrid
allow
SentiBot
ai assistant
SentiOne
SentiBot
observe
Sentry
security
Sentry
Sentry
allow
Sentry Uptime Monitoring
monitoring
Sentry Uptime Bot
Sentry
allow
SentryUptimeBot
ai coding agent
Sentry
SentryUptimeBot
observe
Senutobot
seo crawler
Senuto
Senutobot
challenge
SEO Audit Check Bot
seo crawler
SEO Audit Check
seo-audit-check-bot
challenge
seo4ajax
seo crawler
Capsule Code
seo4ajax
challenge
seo4ajax.com
seo crawler
Capsule Code
seo4ajax.com
challenge
SeobilityBot
seo crawler
Seobility
SeobilityBot
challenge
SeoCherryBot
other
SeoCherryBot
observe
SEOkicks
seo crawler
SEOkicks
SEOkicks
challenge
SEOkicks-Robot
other
SEOkicks-Robot
observe
SEOlizer
seo crawler
SEOLizer
SEOlizer
challenge
SeoLyt
other
SeoLyt
observe
seoscanners
other
seoscanners
observe
seostar.co
other
seostar.co
observe
SEOstats
other
SEOstats
observe
SequelWP
monitoring
SequelWP
SequelWP
allow
SERankingBacklinksBot
seo crawler
SE Ranking
SERankingBacklinksBot
challenge
serendeputybot
ai assistant
Serendeputy
serendeputybot
observe
SerpMinerBot
seo crawler
Serp Miner
SerpMinerBot
challenge
SerpstatBot
seo crawler
Serpstat
serpstatbot
challenge
Server
ai coding agent
StackPath
Server
observe
Server Density
monitoring
Stackpath
allow
ServerHunterSpider
other
Server Hunter
ServerHunterSpider
observe
Seznam
search engine
Seznam
Seznam
allow
SeznamBot
search engine
Seznam
SeznamBot
allow
SFDC-Callout
ai assistant
Salseforce
SFDC-Callout
observe
Shap-User
ai assistant
Parallel
Shap-User
observe
ShapBot
ai crawler
Parallel
ShapBot
observe
ShellBot
other
ShellBot
observe
Shopify-Captain-Hook
other
Shopify
allow
ShortPixel
ai assistant
ShortPixel
ShortPixel
observe
Shortwave Image Fetcher
social media
Shortwave Communications Inc.
Shortwave
allow
Sidetrade
other
Sidetrade
observe
Silktide
ai coding agent
Silktide
Silktide
observe
SilktideBot
other
Silktide
SilktideBot
allow
Silvestro
ai assistant
Silverstream AI
observe
signed
SimpleAsyncPageCrawler
other
SimpleAsyncPageCrawler
observe
simplecrawler
other
simplecrawler
observe
SimplePie
ai assistant
SimplePie
SimplePie
observe
SimpleScraper
other
SimpleScraper
observe