Updated daily

Crawler Catalog

2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

497

Allowed

1,426

Observed

191

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

All 2,114 Search Engine 134 AI Crawler 72 AI Assistant 341 AI Agent 22 AI Coding Agent 77 Browser Automation 2 AI Data Provider 22 AI Scraper 45 SEO Crawler 193 Social Media 73 Security 82 Monitoring 122 Archiver 21 Other 908

2,114 results

Zendesk Webhook other

Zendesk

ZenphonyBot ai agent

Zenphony

Zeus Link Scout other

ZipchatBot ai assistant

Zipchat Inc

Zoombot seo crawler

SEOZoom

ZoomInfo search engine

ZoomInfo

ZoteroTranslationServer ai assistant

Wikimedia

ZoteroTranslationServer

ZumBot search engine

Zum Internet Corp

ZuperlistBot other

Zvelo monitoring

zvelo

← Prev 36 / 36 Next →