Skip to main content
Updated daily

Crawler Catalog

1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

460

Allowed

1,224

Observed

192

Challenged

0

Blocked
Clear
All 1,876 Search Engine 114 AI Crawler 64 AI Assistant 268 AI Agent 10 AI Coding Agent 75 Browser Automation 3 AI Data Provider 13 AI Scraper 40 SEO Crawler 187 Social Media 73 Security 74 Monitoring 122 Archiver 21 Other 812

187 results · SEO Crawler

ParticleNewsBot seo crawler
challenge
PinterestBot seo crawler

Pinterest

Pinterest
challenge
prerender seo crawler

Prerender, LLC

Prerender
challenge
Pricey seo crawler

Wilico

challenge signed
Pro seo crawler

Pro-Sitemaps

Pro
challenge
Productsup Crawler seo crawler

Productsup

challenge
Proximic seo crawler

ComScore

proximic
challenge
Quantcastbot seo crawler

Quantcast

Quantcastbot
challenge
Quartr Crawler seo crawler

Quartr

challenge
RankMath seo crawler

Rank Math

RankMath
challenge
RavenCrawler seo crawler

Raven

RavenCrawler
challenge
Reelevant seo crawler

Reelevant

Reelevant
challenge
Revvim seo crawler

Revvim

challenge
RevvimGort seo crawler

Revvim

RevvimGort
challenge
rogerbot-crawler seo crawler

Moz

rogerbot-crawler
challenge
RSiteAuditor seo crawler

DataForSEO

RSiteAuditor
challenge
RyteBot seo crawler

Ryte

RyteBot
challenge
Salesforce seo crawler

Salesforce

challenge
SalesViewerBot seo crawler

SalesViewer GmbH

SalesViewerBot
challenge
Scrapy seo crawler

Scrapy

Scrapy
challenge
Screaming seo crawler

Screaming Frog

Screaming
challenge
SearchAtlas Bot seo crawler

Search Atlas

SearchAtlas
challenge
SEBot-WA seo crawler

SE Ranking

SEBot-WA
challenge
Semrushbot seo crawler

Semrush

SemrushBot
challenge
SemrushBot-BA seo crawler

Semrush

SemrushBot-BA
challenge
SemrushBot-CT seo crawler

Semrush

SemrushBot-CT
challenge
SemrushBot-SA seo crawler

Semrush

SemrushBot-SA
challenge
SemrushBotBacklinkAudit seo crawler

Semrush

challenge
SemrushBotBacklinks seo crawler

Semrush

SemrushBotBacklinks
challenge
SemrushBotLinkBuilding seo crawler

Semrush

challenge
SemrushBotSI seo crawler

Semrush

SemrushBot-SI
challenge
Senutobot seo crawler

Senuto

Senutobot
challenge
SEO Audit Check Bot seo crawler

SEO Audit Check

seo-audit-check-bot
challenge
seo4ajax seo crawler

Capsule Code

seo4ajax
challenge
seo4ajax.com seo crawler

Capsule Code

seo4ajax.com
challenge
SeobilityBot seo crawler

Seobility

SeobilityBot
challenge
SEOkicks seo crawler

SEOkicks

SEOkicks
challenge
SEOlizer seo crawler

SEOLizer

SEOlizer
challenge
SERankingBacklinksBot seo crawler

SE Ranking

SERankingBacklinksBot
challenge
SerpMinerBot seo crawler

Serp Miner

SerpMinerBot
challenge
SerpstatBot seo crawler

Serpstat

serpstatbot
challenge
SiteAuditBot seo crawler

Semrush

SiteAuditBot
challenge
Sitebulb seo crawler

Sitebulb

Sitebulb
challenge
SiteCheckerBotCrawler seo crawler

Sitechecker

SiteCheckerBotCrawler
challenge
SiteGuruCrawler seo crawler

SiteGuru

SiteGuruCrawler
challenge
Siteimprove seo crawler

Siteimprove

Siteimprove
challenge
Siteimprove Crawl seo crawler

Siteimprove

SiteCheck-sitecrawl
challenge
Siteliner seo crawler

Siteliner

Siteliner
challenge
Slickstream seo crawler

Slickstream

slickstream
challenge
SmartologyBot seo crawler

Smartology

SmartologyBot
challenge
Statabot seo crawler

StataCorp LLC

Statabot
challenge
StatsDroneBot seo crawler

StatsDrone

StatsDroneBot
challenge
StatusNestBacklinkSpider seo crawler

StatusNest

StatusNestBacklinkSpider
challenge
Taboola seo crawler

Taboola

Taboola
challenge
Trellis-Services seo crawler

Mediavine

Trellis-Services
challenge
TTD Content seo crawler

The Trade Desk

TTD-Content
challenge
upday seo crawler

upday GmbH & Co. KG

upday
challenge
videootv Bot seo crawler

Digital Green

videootvBot
challenge
W3 Validator Services seo crawler

World Wide Web Consortium (W3C)

W3
challenge
WebSpiderMount seo crawler

AspenTechLabs Inc

webspidermount
challenge
← Prev 3 / 4 Next →