Updated daily

Crawler Catalog

1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

471

Allowed

1,284

Observed

198

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

Clear

All 1,953 Search Engine 118 AI Crawler 69 AI Assistant 290 AI Agent 11 AI Coding Agent 75 Browser Automation 2 AI Data Provider 13 AI Scraper 41 SEO Crawler 193 Social Media 73 Security 81 Monitoring 122 Archiver 21 Other 844

11 results · AI Agent

AmazonBuyForMe ai agent

Amazon

AmazonBuyForMe

anthropic-ai ai agent

Anthropic

Aranet-SearchBot ai agent

Aranet-SearchBot

Claude-Web ai agent

Anthropic

cohere-ai ai agent

Cohere

iAskBot ai agent

iAsk

iaskspider ai agent

Ai Search

KunatoCrawler ai agent

Zzazz

KunatoCrawler

NovaAct ai agent

Amazon

WRTNBot ai agent

Wrtn

ZenphonyBot ai agent

Zenphony