Updated daily

Crawler Catalog

1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

471

Allowed

1,284

Observed

198

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

All 1,953 Search Engine 118 AI Crawler 69 AI Assistant 290 AI Agent 11 AI Coding Agent 75 Browser Automation 2 AI Data Provider 13 AI Scraper 41 SEO Crawler 193 Social Media 73 Security 81 Monitoring 122 Archiver 21 Other 844

1,953 results

Yelp

Ahrefs

Naver

Yeti by Naver search engine

Naver

Yext

YGS Group Falconer Scraper ai crawler

ygs-scraper-bot

observe

YisouSpider search engine

Yisou

YisouSpider

allow

YiyanBot ai assistant

Baidu

YiyanBot

observe

Yokoy Group Webhooks other

Yokoy Group AG

You.com

ZanistaBot ai crawler

Zanista

Zapier Inc.

Zendesk Webhook other

Zendesk

Zenphony

Zeus Link Scout other

ZipchatBot ai assistant

Zipchat Inc

SEOZoom

ZoomInfo search engine

ZoomInfo

ZoominfoBot

allow

ZoteroTranslationServer ai assistant

Wikimedia

ZoteroTranslationServer

Zum Internet Corp

zvelo

← Prev 33 / 33 Next →