Updated daily

Crawler Catalog

1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

460

Allowed

1,224

Observed

192

Challenged

0

Blocked

Clear

All 1,876 Search Engine 114 AI Crawler 64 AI Assistant 268 AI Agent 10 AI Coding Agent 75 Browser Automation 3 AI Data Provider 13 AI Scraper 40 SEO Crawler 187 Social Media 73 Security 74 Monitoring 122 Archiver 21 Other 812

10 results · AI Agent

anthropic-ai ai agent

Anthropic

Aranet-SearchBot ai agent

Aranet-SearchBot

Claude-Web ai agent

Anthropic

cohere-ai ai agent

Cohere

iAskBot ai agent

iAsk

iaskspider ai agent

Ai Search

KunatoCrawler ai agent

Zzazz

KunatoCrawler

NovaAct ai agent

Amazon

WRTNBot ai agent

Wrtn

ZenphonyBot ai agent

Zenphony