Updated daily
Crawler Catalog
1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
1,876 results
RevvimGort
seo crawler
Revvim
RevvimGort
challenge
Reward Gateway
security
Reward Gateway
reward-gateway
allow
RidderBot
other
RidderBot
observe
Riddler
other
Riddler
observe
Rivva
other
Rivva
observe
Robinbot
monitoring
Robin Education
allow
signed
robots-ai-permissions
other
robots-ai-permissions
observe
Robozilla
other
Robozilla
observe
rogerbot-crawler
seo crawler
Moz
rogerbot-crawler
challenge
RSiteAuditor
seo crawler
DataForSEO
RSiteAuditor
challenge
RSS API
social media
RSS API (by Tibush GmbH)
RSSAPI
allow
rss-is-dead.lol
other
rss-is-dead.lol
observe
RSS.Social
ai assistant
RSS.Social
RSS.Social
observe
rss2tg_bot
social media
Yellow Rubber Duck Consulting
rss2tg
allow
RSSBot
other
RSSBot
observe
RSSingBot
other
RSSingBot
observe
RukiCrawler
other
RukiCrawler
observe
RuxitSynthetic
other
RuxitSynthetic
observe
RyeBot
ai assistant
Rye
observe
signed
RyteBot
seo crawler
Ryte
RyteBot
challenge
s4a
other
s4a
observe
SaaSBrowserBot
other
SaaSBrowserBot
observe
saasworksbot
other
saasworksbot
observe
SafeDNSBot
other
SafeDNSBot
observe
SafeSearch
other
SafeSearch
observe
SalamandraVLM
ai scraper
Aithlas
SalamandraVLM
observe
Salesforce
seo crawler
Salesforce
challenge
SalesViewerBot
seo crawler
SalesViewer GmbH
SalesViewerBot
challenge
Sansec Security Monitor
security
Sansec Security Monitor
Sansec
allow
SBIntuitionsBot
ai scraper
SB Intuitions
SBIntuitionsBot
observe
SBL-BOT
other
SBL-BOT
observe
ScaleUpXQ-Crawler
other
ScaleUpXQ-Crawler
observe
ScavoBot
ai assistant
Scavo
observe
signed
Schema-Markup-Validator
other
Schema-Markup-Validator
observe
scoop.it
ai assistant
Meltwater
scoop.it
observe
scoopit-crawler
ai assistant
Scoop.it
scoopit-crawler
observe
Scope3
ai assistant
Scope3
Scope3
observe
score3
other
score3
observe
Scour RSS Bot
social media
Scour
allow
scour.ing
ai assistant
Scour
scour.ing
observe
ScourRSSBot
ai assistant
Scour
ScourRSSBot
observe
ScoutJet
other
ScoutJet
observe
[email protected]
ai assistant
The New York Times
[email protected]
observe
Scrapy
seo crawler
Scrapy
Scrapy
challenge
Screaming
seo crawler
Screaming Frog
Screaming
challenge
scribdbot
other
scribdbot
observe
Scrubby
search engine
Scrub the Web
Scrubby
allow
Scrunchbot
ai coding agent
Scrunch
Scrunchbot
observe
SE Ranking Bot
monitoring
SE Ranking
SE
allow
search.marginalia.nu
search engine
Marginalia
search.marginalia.nu
allow
SearchAtlas Bot
seo crawler
Search Atlas
SearchAtlas
challenge
SearchmetricsBot
other
SearchmetricsBot
observe
searchpreview
other
searchpreview
observe
SEBot-WA
seo crawler
SE Ranking
SEBot-WA
challenge
Security-Audit
other
Security-Audit
observe
SecurityHeaders
security
Security Headers
SecurityHeaders
allow
Seekbot
other
Seek Bot
Seekbot
observe
Seekport
search engine
SISTRIX
Seekport
allow
SeekportBot
search engine
SISTRIX
SeekportBot
allow
Seekr
other
Seekr
observe