Updated daily

Crawler Catalog

2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

497

Allowed

1,426

Observed

191

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

Clear

All 2,114 Search Engine 134 AI Crawler 72 AI Assistant 341 AI Agent 22 AI Coding Agent 77 Browser Automation 2 AI Data Provider 22 AI Scraper 45 SEO Crawler 193 Social Media 73 Security 82 Monitoring 122 Archiver 21 Other 908

908 results · Other

CyberFindCrawler other

CyberFindCrawler

CyberPatrol other

CyotekWebCopy other

CyotekWebCopy

d-bot-health-check other

d-bot-health-check

DareBoost other

Dark Visitor other

Datafeedwatch other

Datafeedwatch

datagnionbot other

Dazzle BlueSky Bot other

Digg Deeper other

Digincore bot other

DittoSpyder other

DnBCrawler-Analytics other

DnBCrawler-Analytics

DNSCrawler other

DNSScanner other

Domain Re-Animator Bot other

DomainCrawler other

DomainCrawler

DomainEvidenceBot other

DomainEvidenceBot

DomainMapper other

Domains Project other

DominicBot other

Dow Jones Searchbot other

Download Ninja other

Drupalbot other

e.ventures Investment Crawler other

EasyBib AutoCite other

EasyBill Import Manager other

easybill.de

easybill-ImportManager

EasyCron

easyDNS Monitoring other

ec2linkfinder other

ec2linkfinder

ecoresearchCrawler other

ecoresearchCrawler

edisterbot other

electricmonk other

ellisphere other

EmailCollector other

EmailCollector

EmailSiphon other

EmailWolf other

eManagerBot other

emetriqContextualBot other

emetriqContextualBot

EriounBot other

Erioun

EroCrawler other

EtaoSpider other

europarchive.org other

europarchive.org

evc-batch other

← Prev 4 / 16 Next →