Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
134 results · Search Engine
Y!J-BRZ/YATSHA crawler
search engine
Yahoo
allow
yacybot
search engine
YaCy
yacybot
allow
Yahoo Slurp
search engine
Yahoo
allow
YandexBot
search engine
Yandex
http://yandex.com/bots
allow
rDNS
YandexImages
search engine
Yandex
YandexImages
allow
rDNS
YandexMobileBot
search engine
Yandex
YandexMobileBot
allow
rDNS
YandexRenderResourcesBot
search engine
Yandex
YandexRenderResourcesBot
allow
rDNS
YandexVertis
search engine
Yandex
YandexVertis
allow
rDNS
Yeti
search engine
Naver
Yeti
allow
Yeti by Naver
search engine
Naver
allow
YisouSpider
search engine
Yisou
YisouSpider
allow
YouBot
search engine
You.com
YouBot
allow
rDNS
ZoomInfo
search engine
ZoomInfo
ZoominfoBot
allow
ZumBot
search engine
Zum Internet Corp
ZumBot
allow