Updated daily

Crawler Catalog

1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

471

Allowed

1,284

Observed

198

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

Clear

All 1,953 Search Engine 118 AI Crawler 69 AI Assistant 290 AI Agent 11 AI Coding Agent 75 Browser Automation 2 AI Data Provider 13 AI Scraper 41 SEO Crawler 193 Social Media 73 Security 81 Monitoring 122 Archiver 21 Other 844

844 results · Other

Dazzle BlueSky Bot other

Digg Deeper other

Digincore bot other

DittoSpyder other

DnBCrawler-Analytics other

DnBCrawler-Analytics

DNSCrawler other

DNSScanner other

Domain Re-Animator Bot other

DomainCrawler other

DomainCrawler

DomainMapper other

Domains Project other

DominicBot other

Dow Jones Searchbot other

Download Ninja other

Drupalbot other

e.ventures Investment Crawler other

EasyBib AutoCite other

EasyBill Import Manager other

easybill.de

easybill-ImportManager

EasyCron

easyDNS Monitoring other

ec2linkfinder other

ec2linkfinder

ecoresearchCrawler other

ecoresearchCrawler

edisterbot other

electricmonk other

ellisphere other

EmailCollector other

EmailCollector

EmailSiphon other

EmailWolf other

emetriqContextualBot other

emetriqContextualBot

EroCrawler other

EtaoSpider other

europarchive.org other

europarchive.org

evc-batch other

everyfeed-spider other

everyfeed-spider

EveryoneSocialBot other

EveryoneSocialBot

Exodus

ExodusMovement

Experibot other

expo-research-bot other

expo-research-bot

ExteContextCrawl other

ExteContextCrawl

ExtLinksBot other

ExtractorPro other

Eyeotabot other

FAB-Scraper other

FairAd Client other

FAST Enterprise Crawler other

← Prev 4 / 15 Next →