Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
191 results
Prerender, LLC
Prerender
Wilico
pricey
Pro-Sitemaps
Productsup
ComScore
proximic
QlyzeBot
Quantcast
Quantcastbot
Quartr
quartr-crawler
Rank Math
RankMath
Raven
RavenCrawler
Reelevant
Reelevant
Revvim
RevvimGort
Revvim
RevvimGort
Moz
rogerbot-crawler
DataForSEO
RSiteAuditor
Ryte
RyteBot
Salesforce
SFDC-Callout
SalesViewer GmbH
SalesViewerBot
Scrapy
Scrapy
Screaming Frog
Screaming
Search Atlas
SearchAtlas
SE Ranking
SEBot-WA
Semrush
SemrushBot
Semrush
SemrushBot-BA
Semrush
SemrushBot-CT
Semrush
SemrushBot-SA
Semrush
SemrushBot-BA
Semrush
SemrushBotBacklinks
Semrush
Semrush
SemrushBot-SI
Senuto
Senutobot
SEO Audit Check
seo-audit-check-bot
Capsule Code
seo4ajax
Capsule Code
seo4ajax.com
Seobility
SeobilityBot
SEOJuice
SEOJuice-SearchBot
SEOkicks
SEOkicks
SEOLizer
SEOlizer
SEOToolsOutreach
SE Ranking
SERankingBacklinksBot
Serp Miner
SerpMinerBot
Serpstat
serpstatbot
Semrush
SiteAuditBot
Sitebulb
Sitebulb
Sitechecker
SiteCheckerBotCrawler
SiteGuru
SiteGuruCrawler
Siteimprove
SiteCheck-sitecrawl
Siteliner
Siteliner
Slickstream
SlickBot
Smartology
SmartologyBot
Spyglasses
SpyglassesBot
Open-source
StataCorp LLC
Statabot
StatsDrone
StatsDroneBot
StatusNest
StatusNestBacklinkSpider
Taboola
Taboolabot
Mediavine
Trellis-Services
The Trade Desk
TTD-Content
upday GmbH & Co. KG
upday
Digital Green
videootvBot