Updated daily

Crawler Catalog

1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

460

Allowed

1,224

Observed

192

Challenged

0

Blocked

Clear

All 1,876 Search Engine 114 AI Crawler 64 AI Assistant 268 AI Agent 10 AI Coding Agent 75 Browser Automation 3 AI Data Provider 13 AI Scraper 40 SEO Crawler 187 Social Media 73 Security 74 Monitoring 122 Archiver 21 Other 812

64 results · AI Crawler

YandexAdditional ai crawler

Yandex

YandexAdditional

YandexAdditionalBot ai crawler

Yandex

YandexAdditionalBot

YGS Group Falconer Scraper ai crawler

ZanistaBot ai crawler

Zanista

← Prev 2 / 2 Next →