Updated daily

Crawler Catalog

1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

471

Allowed

1,284

Observed

198

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

Clear

All 1,953 Search Engine 118 AI Crawler 69 AI Assistant 290 AI Agent 11 AI Coding Agent 75 Browser Automation 2 AI Data Provider 13 AI Scraper 41 SEO Crawler 193 Social Media 73 Security 81 Monitoring 122 Archiver 21 Other 844

69 results · AI Crawler

TalarionSearchBot ai crawler

Talarion

TalarionSearchBot

TavilyBot ai crawler

Tavily

WARDBot ai crawler

WEBSPARK

xAI-SearchBot ai crawler

xAI

xAI-SearchBot

YandexAdditional ai crawler

Yandex

YandexAdditional

YandexAdditionalBot ai crawler

Yandex

YandexAdditionalBot

YepBot ai crawler

Ahrefs

YGS Group Falconer Scraper ai crawler

ygs-scraper-bot

ZanistaBot ai crawler

Zanista

← Prev 2 / 2 Next →