Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
2,114 results
Semantic Visions
semantic-visions-analytica
semantic-visions-discovery
Semantic Visions
semantic-visions-infoseeker
semantic-visions.com
Semanticbot
Ai2
SemanticScholarBot
Semantic Visions
SemanticVisionsBot
sempi.tech
Semrush
SemrushBot
Semrush
SemrushBot-BA
SemrushBot-BM
Semrush
SemrushBot-CT
Semrush
SemrushBot-OCOB
Semrush
SemrushBot-SA
Semrush
SemrushBot-BA
Semrush
SemrushBotBacklinks
Semrush
Semrush
SemrushBot-SI
Semrush
SemrushBot-SWA
Sender Online Marketing
SenderBot
SendGrid
SentiOne
SentiBot
Sentry
https://sentry.io
Sentry Uptime Bot
Sentry
Sentry
SentryUptimeBot
Senuto
Senutobot
SEO Audit Check
seo-audit-check-bot
Capsule Code
seo4ajax
Capsule Code
seo4ajax.com
Seobility
SeobilityBot
SeoCherryBot
SEOJuice
SEOJuice-SearchBot
SEOkicks
SEOkicks
SEOkicks-Robot
SEOLizer
SEOlizer
SeoLyt
seoscanners
seostar.co
SEOstats
SEOToolsOutreach
SequelWP
SequelWP
Nexigen Digital
SE Ranking
SERankingBacklinksBot
Serendeputy
serendeputybot
Serp Miner
SerpMinerBot
Serpstat
serpstatbot
Stackpath
Server
StackPath
Server Hunter
ServerHunterSpider
Seznam
Android(10);cz.seznam.sbrowser;12.6.0
Seznam
SeznamBot
Salseforce
SFDC-Callout
Parallel
Shap-User
Parallel
ShapBot
ShellBot
Shopify
Shopify-Captain-Hook
ShortPixel
ShortPixel
Shortwave Communications Inc.
Shortwave
Sidetrade
Silktide
Silktide