Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
497 results
YahooCacheSystem
social media
Yahoo
YahooCacheSystem
allow
YahooMailProxy
social media
Yahoo
YahooMailProxy
allow
YandexBot
search engine
Yandex
http://yandex.com/bots
allow
rDNS
YandexImages
search engine
Yandex
YandexImages
allow
rDNS
YandexMobileBot
search engine
Yandex
YandexMobileBot
allow
rDNS
YandexRenderResourcesBot
search engine
Yandex
YandexRenderResourcesBot
allow
rDNS
YandexVertis
search engine
Yandex
YandexVertis
allow
rDNS
Yeti
search engine
Naver
Yeti
allow
Yeti by Naver
search engine
Naver
allow
YisouSpider
search engine
Yisou
YisouSpider
allow
Yokoy Group Webhooks
other
Yokoy Group AG
allow
YouBot
search engine
You.com
YouBot
allow
rDNS
Zapier
other
Zapier Inc.
Zapier
allow
Zendesk Webhook
other
Zendesk
allow
ZoomInfo
search engine
ZoomInfo
ZoominfoBot
allow
ZumBot
search engine
Zum Internet Corp
ZumBot
allow
Zvelo
monitoring
zvelo
zvelo-bot
allow