Updated daily

Crawler Catalog

1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

471

Allowed

1,284

Observed

198

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

Clear

All 1,953 Search Engine 118 AI Crawler 69 AI Assistant 290 AI Agent 11 AI Coding Agent 75 Browser Automation 2 AI Data Provider 13 AI Scraper 41 SEO Crawler 193 Social Media 73 Security 81 Monitoring 122 Archiver 21 Other 844

1,284 results

Yellowbrandprotectionbot other

Yellowbrandprotectionbot

Yelp

Ahrefs

YGS Group Falconer Scraper ai crawler

ygs-scraper-bot

observe

YiyanBot ai assistant

Baidu

ZanistaBot ai crawler

Zanista

Zenphony

Zeus Link Scout other

ZipchatBot ai assistant

Zipchat Inc

ZoteroTranslationServer ai assistant

Wikimedia

ZoteroTranslationServer

← Prev 22 / 22 Next →