Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
2,114 results
Moz
rogerbot-crawler
DataForSEO
RSiteAuditor
RSS API (by Tibush GmbH)
RSSAPI
rss-is-dead.lol
RSS.Social
RSS.Social
Yellow Rubber Duck Consulting
rss2tg
RSSBot
RSSExtractor
RSSingBot
RukiCrawler
RuxitSynthetic
Rye
ryebot
Ryte
RyteBot
s4a
SaaSBrowserBot
saasworksbot
SafeDNSBot
SafeSearch
Aithlas
SalamandraVLM
Salesforce
SFDC-Callout
SalesViewer GmbH
SalesViewerBot
SaluteBot
Sansec Security Monitor
Sansec
SB Intuitions
SBIntuitionsBot
SBL-BOT
ScaleUpXQ-Crawler
Scalinth
ScalinthBot
ScannerBot
Scavo
ScavoBot
Schema-Markup-Validator
Meltwater
scoop.it
Scoop.it
scoopit-crawler
Scope3
Scope3
score3
Scour
scour.ing
Scour
scour.ing
Scour
ScourRSSBot
ScoutJet
The New York Times
[email protected]
Scrapy
Scrapy
Screaming Frog
Screaming
scribdbot
Scrub the Web
Scrubby
Scrunch
Scrunchbot
Unflatten
ScryBot
SE Ranking
SE
Marginalia
search.marginalia.nu
Search Atlas
SearchAtlas
SearchmetricsBot
searchpreview
SE Ranking
SEBot-WA
Security-Audit
Security Headers
SecurityHeaders
SecurityResearch
Seek Bot
Seekbot
SISTRIX
Seekport
SISTRIX
SeekportBot
Seekr
seewithkids
SeltzSitemapFetcher