Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,953 results
SemrushBot-BM
Semrush
SemrushBot-CT
Semrush
SemrushBot-OCOB
Semrush
SemrushBot-SA
Semrush
SemrushBot-SI
Semrush
SemrushBot-SWA
Semrush
SemrushBot-BA
Semrush
SemrushBotBacklinks
Semrush
Semrush
SemrushBot-SI
Semrush
SemrushBot-SWA
SendGrid
SentiOne
SentiBot
Sentry
https://sentry.io
Sentry Uptime Bot
Sentry
Sentry
SentryUptimeBot
Senuto
Senutobot
SEO Audit Check
seo-audit-check-bot
Capsule Code
seo4ajax
Capsule Code
seo4ajax.com
Seobility
SeobilityBot
SeoCherryBot
SEOkicks
SEOkicks
SEOkicks-Robot
SEOLizer
SEOlizer
SeoLyt
seoscanners
seostar.co
SEOstats
SequelWP
SequelWP
Nexigen Digital
SE Ranking
SERankingBacklinksBot
Serendeputy
serendeputybot
Serp Miner
SerpMinerBot
Serpstat
serpstatbot
Serpstat
serpstatbot
Stackpath
Server
StackPath
Server Hunter
ServerHunterSpider
Seznam
Android(10);cz.seznam.sbrowser;12.6.0
Seznam
SeznamBot
Salseforce
SFDC-Callout
Parallel
Shap-User
Parallel
ShapBot
ShellBot
Shopify
Shopify-Captain-Hook
ShortPixel
ShortPixel
Shortwave Communications Inc.
Shortwave
Sidetrade
Silktide
Silktide
Silktide
SilktideBot
Silverstream AI
silvestro
SimpleAsyncPageCrawler
simplecrawler
SimplePie
SimplePie
SimpleScraper
Sindup
Sirdata
SirdataBot
sistrix
SISTRIX GmbH