Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
2,114 results
xAI-SearchBot
ai crawler
xAI
xAI-SearchBot
observe
Xbot
other
Xbot
observe
xenU
other
xenU
observe
Xenu Link Sleuth
other
observe
Xenu's
other
Xenu's
observe
Xenu's Link Sleuth 1.1c
other
Xenu&
observe
Xing Bot
other
Xing
observe
XML Sitemaps Generator
other
observe
xovibot
other
xovibot
observe
XoviOnpageCrawler
seo crawler
XOVI
XoviOnpageCrawler
challenge
Xpanse-bot
other
Xpanse-bot
observe
XY Archive Compliance Bot
other
XY Archive Compliance
XY-Archive-Compliance
observe
XY-Archive-Compliance-Archiver
archiver
XY Planning Network
XY-Archive-Compliance-Archiver
allow
Y!J-ASR
other
Y!J-ASR
observe
Y!J-BRW
other
Y!J-BRW
observe
Y!J-BRZ/YATSHA crawler
search engine
Yahoo
allow
Y!J-DLC
other
Y!J-DLC
observe
yacybot
search engine
YaCy
yacybot
allow
YaDirectFetcher
ai assistant
Yandex
YaDirectFetcher
observe
Yahoo Ad Monitoring
seo crawler
Yahoo
challenge
Yahoo Japan
seo crawler
Yahoo Japan Corporation
Yahoo
challenge
Yahoo Japan SEO Crawler
seo crawler
Yahoo
J-BRW
challenge
Yahoo Link Preview
social media
Yahoo
Yahoo Link Preview
allow
Yahoo Mail
social media
Yahoo
YahooMailProxy
allow
Yahoo Pipes 1.0
other
observe
Yahoo Slurp
search engine
Yahoo
allow
YahooCacheSystem
social media
Yahoo
YahooCacheSystem
allow
YahooMailProxy
social media
Yahoo
YahooMailProxy
allow
YaK
ai assistant
Linkfluence
YaK
observe
YandexAdditional
ai crawler
Yandex
YandexAdditional
observe
YandexAdditionalBot
ai crawler
Yandex
YandexAdditionalBot
observe
YandexBot
search engine
Yandex
http://yandex.com/bots
allow
rDNS
YandexImages
search engine
Yandex
YandexImages
allow
rDNS
YandexMobileBot
search engine
Yandex
YandexMobileBot
allow
rDNS
YandexRenderResourcesBot
search engine
Yandex
YandexRenderResourcesBot
allow
rDNS
YandexVertis
search engine
Yandex
YandexVertis
allow
rDNS
YandexVideo
other
YandexVideo
observe
YandoriRSSBot
other
YandoriRSSBot
observe
yanga
other
yanga
observe
Yellowbrandprotectionbot
other
Yellowbrandprotectionbot
observe
YelpBot
ai assistant
Yelp
YelpBot
observe
YepBot
ai crawler
Ahrefs
YepBot
observe
Yeti
search engine
Naver
Yeti
allow
Yeti by Naver
search engine
Naver
allow
Yext Inc
seo crawler
Yext
YextBot
challenge
YGS Group Falconer Scraper
ai crawler
ygs-scraper-bot
observe
YiBot
ai scraper
01.AI
YiBot
observe
YisouSpider
search engine
Yisou
YisouSpider
allow
YiyanBot
ai assistant
Baidu
YiyanBot
observe
Yokoy Group Webhooks
other
Yokoy Group AG
allow
yoozBot
other
yoozBot
observe
YouBot
search engine
You.com
YouBot
allow
rDNS
YoudaoBot
other
YoudaoBot
observe
Youmag
other
Youmag
observe
Zabbix
other
Zabbix
observe
ZanistaBot
ai crawler
Zanista
ZanistaBot
observe
Zao
other
Zao
observe
Zapier
other
Zapier Inc.
Zapier
allow
Zealbot
other
Zealbot
observe
zenback bot
other
zenback
observe