Updated daily

Crawler Catalog

2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

497

Allowed

1,426

Observed

191

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

Clear

All 2,114 Search Engine 134 AI Crawler 72 AI Assistant 341 AI Agent 22 AI Coding Agent 77 Browser Automation 2 AI Data Provider 22 AI Scraper 45 SEO Crawler 193 Social Media 73 Security 82 Monitoring 122 Archiver 21 Other 908

22 results · AI Agent

anthropic-ai ai agent

Anthropic

Aranet-SearchBot ai agent

Aranet-SearchBot

CartAI Agentic Commerce ai agent

CartAI

Claude-Web ai agent

Anthropic

CledaraBot ai agent

Cledara

cohere-ai ai agent

Cohere

OpenAI

Grok Bot ai agent

xAI

grok-agent ai agent

xAI

GrokBot ai agent

xAI

HarkBot ai agent

Hark

iAskBot ai agent

iAsk

iaskspider ai agent

Ai Search

Instinct ai agent

Instinct

Kimi-Agent ai agent

Moonshot AI

KunatoCrawler ai agent

Zzazz

KunatoCrawler

Meta

NovaAct ai agent

Amazon

SomethinggBot ai agent

somethingg

SomethinggBot

WRTNBot ai agent

Wrtn

xAI-Grok ai agent

xAI

ZenphonyBot ai agent

Zenphony