Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
908 results · Other
research/recap
other
research/recap
observe
Retool
other
Retool
Retool
observe
RetrevoPageAnalyzer
other
RetrevoPageAnalyzer
observe
RetroDocumentResearch
other
RetroDocumentResearch
observe
ReverseEngineeringBot
other
ReverseEngineeringBot
observe
RevicBot
other
RevicBot
observe
RidderBot
other
RidderBot
observe
Riddler
other
Riddler
observe
Rivva
other
Rivva
observe
robots-ai-permissions
other
robots-ai-permissions
observe
Robozilla
other
Robozilla
observe
rss-is-dead.lol web bot
other
rss-is-dead.lol
observe
RSSBot
other
RSSBot
observe
RSSingBot
other
RSSingBot
observe
RukiCrawler
other
RukiCrawler
observe
RuxitSynthetic
other
RuxitSynthetic
observe
s4a
other
s4a
observe
SaaSBrowserBot
other
SaaSBrowserBot
observe
saasworksbot
other
saasworksbot
observe
SafeDNSBot
other
SafeDNSBot
observe
SafeSearch microdata crawler
other
SafeSearch
observe
SaluteBot
other
SaluteBot
observe
SBL-BOT
other
SBL-BOT
observe
ScaleUpXQ-Crawler
other
ScaleUpXQ-Crawler
observe
ScannerBot
other
ScannerBot
observe
Schema-Markup-Validator
other
Schema-Markup-Validator
observe
score3
other
score3
observe
ScoutJet
other
ScoutJet
observe
scribdbot
other
scribdbot
observe
SearchmetricsBot
other
SearchmetricsBot
observe
searchpreview
other
searchpreview
observe
Security-Audit
other
Security-Audit
observe
SecurityResearch
other
SecurityResearch
observe
Seekbot
other
Seek Bot
Seekbot
observe
Seekr
other
Seekr
observe
seewithkids
other
seewithkids
observe
SeltzSitemapFetcher
other
SeltzSitemapFetcher
observe
semantic-visions-discovery
other
semantic-visions-discovery
observe
semantic-visions.com crawler
other
semantic-visions.com
observe
Semanticbot
other
Semanticbot
observe
sempi.tech
other
sempi.tech
observe
SemrushBot-BM
other
SemrushBot-BM
observe
SendGrid
other
SendGrid
allow
SeoCherryBot
other
SeoCherryBot
observe
SEOkicks-Robot
other
SEOkicks-Robot
observe
SeoLyt
other
SeoLyt
observe
seoscanners
other
seoscanners
observe
seostar.co
other
seostar.co
observe
SEOstats
other
SEOstats
observe
ServerHunterSpider
other
Server Hunter
ServerHunterSpider
observe
ShellBot
other
ShellBot
observe
Shopify-Captain-Hook
other
Shopify
Shopify-Captain-Hook
allow
Sidetrade indexer bot
other
Sidetrade
observe
SilktideBot
other
Silktide
SilktideBot
allow
SimpleAsyncPageCrawler
other
SimpleAsyncPageCrawler
observe
simplecrawler
other
simplecrawler
observe
SimpleScraper
other
SimpleScraper
observe
Sindup
other
Sindup
observe
SirdataBot
other
Sirdata
SirdataBot
observe
sistrix crawler
other
sistrix
observe