Updated daily

Crawler Catalog

2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

497

Allowed

1,426

Observed

191

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

Clear

All 2,114 Search Engine 134 AI Crawler 72 AI Assistant 341 AI Agent 22 AI Coding Agent 77 Browser Automation 2 AI Data Provider 22 AI Scraper 45 SEO Crawler 193 Social Media 73 Security 82 Monitoring 122 Archiver 21 Other 908

72 results · AI Crawler

SemrushBotSwa ai crawler

Semrush

SemrushBot-SWA

ShapBot ai crawler

Parallel

Spawning AI ai crawler

Spawning

TalarionSearchBot ai crawler

Talarion

TalarionSearchBot

TavilyBot ai crawler

Tavily

WARDBot ai crawler

WEBSPARK

xAI-SearchBot ai crawler

xAI

xAI-SearchBot

YandexAdditional ai crawler

Yandex

YandexAdditional

YandexAdditionalBot ai crawler

Yandex

YandexAdditionalBot

YepBot ai crawler

Ahrefs

YGS Group Falconer Scraper ai crawler

ygs-scraper-bot

ZanistaBot ai crawler

Zanista

← Prev 2 / 2 Next →