Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,953 results
DataForSEO
RSiteAuditor
RSS API (by Tibush GmbH)
RSSAPI
rss-is-dead.lol
RSS.Social
RSS.Social
Yellow Rubber Duck Consulting
rss2tg
RSSBot
RSSingBot
RukiCrawler
RuxitSynthetic
Rye
ryebot
Ryte
RyteBot
s4a
SaaSBrowserBot
saasworksbot
SafeDNSBot
SafeSearch
Aithlas
SalamandraVLM
Salesforce
SFDC-Callout
SalesViewer GmbH
SalesViewerBot
Sansec Security Monitor
Sansec
SB Intuitions
SBIntuitionsBot
SBL-BOT
ScaleUpXQ-Crawler
Scavo
ScavoBot
Schema-Markup-Validator
Meltwater
scoop.it
Scoop.it
scoopit-crawler
Scope3
Scope3
score3
Scour
scour.ing
Scour
scour.ing
Scour
ScourRSSBot
ScoutJet
The New York Times
[email protected]
Scrapy
Scrapy
Screaming Frog
Screaming
scribdbot
Scrub the Web
Scrubby
Scrunch
Scrunchbot
SE Ranking
SE
Marginalia
search.marginalia.nu
SearchAtlas
Search Atlas
SearchAtlas
SearchmetricsBot
searchpreview
SE Ranking
SEBot-WA
Security-Audit
Security Headers
SecurityHeaders
Seek Bot
Seekbot
SISTRIX
Seekport
SISTRIX
SeekportBot
Seekr
seewithkids
semantic-visions-discovery
semantic-visions.com
Semanticbot
Ai2
SemanticScholarBot
sempi.tech
Semrush
SemrushBot
Semrush
SemrushBot-BA