Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
198 results
80legs
008
AgencyAnalytics
aa
Invicti Security
Acunetix
Addsearch
AddSearchBot
AddThis
AddThis
OneTag Limited
AdsTxtCrawler
Agency Analytics
Ahrefs
AhrefsSiteAudit
Ahrefs
AhrefsBot
Alli AI
Amazon
AmazonAdBot
Amazon
contxbot
Attracta
Attracta
1h4x.com
Babbar
Barkrowler
BestChange
BestChange Bot
BigUpData
BigUpDataBot
Microsoft
adidxbot
WebMeUp
BLEXBot
KP Software Consult ApS
BoardGamePrices
Botify
botify
BrightEdge
BrightEdge
Conductor
Caliperbot
Capital One
CapitalOneBot
Channable
channable/importer
Channel3
Channel3Bot
Chrome-Lighthouse
seoClarity
ClarityBot
Mushi Labs
Clearscopebot
Clickagy
Clickagy Intelligence Bot v2
Cloudflare
Cloudflare
Cloudflare
Cloudflare-Healthchecks
Cloudflare
CloudFlare-Prefetch
Cloudflare
Cloudflare
Cloudflare SpeedTest
Cloudflare
Cloudflare-SSLDetector
Cloudflare
Cloudflare Stream Webhook
Cloudflare-Healthchecks
Cloudflare-SSLDetector
Cloudflare
Cloudflare-Traffic-Manager
Cludo
cludo.com
Cocolyze
Cocolyzebot
cognitive SEO Internet Marketing Tools.
cognitiveSEO
Outcomes
ContextualBot
Convermax
Convermax
Criteo
CriteoBot
Cxense
cXensebot
DataForSEO
DataForSEO
DataForSEO
RankMath Link Checker
DataForSEO
DataForSeoBot
dcrawl
Lumar
deepcrawl.com
DigiCert
Digicert
dlvr.it
Game Lounge
DomainStatsBot
Dragon Metrics
Dragonbot
eMoney Advisor
eMoneyBot
Techmeme