Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
193 results · SEO Crawler
Oncrawl
www.oncrawl.com
Oncrawl
Online Webceo
online-webceo-bot
Openindex
OpenindexSpider
Outbrain
outbrain
particlenewsbot
Pinterest
Prerender, LLC
Prerender
Wilico
pricey
Pro-Sitemaps
Productsup
ComScore
proximic
Quantcast
Quantcastbot
Quartr
quartr-crawler
Rank Math
RankMath
Raven
RavenCrawler
Reelevant
Reelevant
Revvim
RevvimGort
Revvim
RevvimGort
Moz
rogerbot-crawler
DataForSEO
RSiteAuditor
Ryte
RyteBot
Salesforce
SFDC-Callout
SalesViewer GmbH
SalesViewerBot
Scrapy
Scrapy
Screaming Frog
Screaming
Search Atlas
SearchAtlas
SE Ranking
SEBot-WA
Semrush
SemrushBot
Semrush
SemrushBot-BA
Semrush
SemrushBot-CT
Semrush
SemrushBot-SA
Semrush
SemrushBot-SI
Semrush
SemrushBot-BA
Semrush
SemrushBotBacklinks
Semrush
Semrush
SemrushBot-SI
Senuto
Senutobot
SEO Audit Check
seo-audit-check-bot
Capsule Code
seo4ajax
Capsule Code
seo4ajax.com
Seobility
SeobilityBot
SEOkicks
SEOkicks
SEOLizer
SEOlizer
SE Ranking
SERankingBacklinksBot
Serp Miner
SerpMinerBot
Serpstat
serpstatbot
Serpstat
serpstatbot
Semrush
SiteAuditBot
Sitebulb
Sitebulb
Sitechecker
SiteCheckerBotCrawler
SiteGuru
SiteGuruCrawler
Siteimprove
SiteCheck-sitecrawl
Siteliner
Siteliner
Slickstream
SlickBot
Smartology
SmartologyBot
StataCorp LLC
Statabot
StatsDrone
StatsDroneBot
StatusNest
StatusNestBacklinkSpider
Taboola
Taboolabot