Updated daily
Crawler Catalog
1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
1,876 results
WSM
other
WSM
observe
WTotem
security
WebTotem
WTotem
allow
WTotemBot
security
WebTotem
WTotemBot
allow
WWW-Collector-E
other
WWW-Collector-E
observe
WWW-Mechanize
other
WWW-Mechanize
observe
www.deadlinkchecker.com
other
www.deadlinkchecker.com
observe
www.uptime.com
other
www.uptime.com
observe
xAI-SearchBot
ai crawler
xAI
xAI-SearchBot
observe
Xbot
other
Xbot
observe
xenU
other
xenU
observe
Xenu's
other
Xenu's
observe
Xing
other
Xing
observe
XML
other
XML
observe
xovibot
other
xovibot
observe
XoviOnpageCrawler
seo crawler
XOVI
XoviOnpageCrawler
challenge
Xpanse-bot
other
Xpanse-bot
observe
XY Archive Compliance Bot
other
XY Archive Compliance
XY-Archive-Compliance
observe
XY-Archive-Compliance-Archiver
archiver
XY Planning Network
XY-Archive-Compliance-Archiver
allow
Y!J-ASR
other
Y!J-ASR
observe
Y!J-BRW
other
Y!J-BRW
observe
Y!J-BRZ/YATSHA
search engine
Yahoo
Y!J-BRZ/YATSHA
allow
Y!J-DLC
other
Y!J-DLC
observe
yacybot
search engine
YaCy
yacybot
allow
YaDirectFetcher
ai assistant
Yandex
YaDirectFetcher
observe
Yahoo
ai assistant
Yahoo
Yahoo
observe
Yahoo Ad Monitoring
seo crawler
Yahoo
challenge
Yahoo Japan
seo crawler
Yahoo Japan Corporation
Yahoo
challenge
Yahoo Japan SEO Crawler
seo crawler
Yahoo
challenge
Yahoo Link Preview
social media
Yahoo
allow
Yahoo Mail
social media
Yahoo
YahooMailProxy
allow
Yahoo Slurp
search engine
Yahoo
allow
YahooCacheSystem
social media
Yahoo
YahooCacheSystem
allow
YahooMailProxy
social media
Yahoo
allow
YaK
ai assistant
Linkfluence
YaK
observe
YandexAdditional
ai crawler
Yandex
YandexAdditional
observe
YandexAdditionalBot
ai crawler
Yandex
YandexAdditionalBot
observe
YandexBot
search engine
Yandex
yandex
allow
YandexImages
search engine
Yandex
YandexImages
allow
YandexMobileBot
search engine
Yandex
YandexMobileBot
allow
YandexRenderResourcesBot
search engine
Yandex
YandexRenderResourcesBot
allow
YandexVertis
search engine
Yandex
YandexVertis
allow
YandexVideo
other
YandexVideo
observe
YandoriRSSBot
other
YandoriRSSBot
observe
yanga
other
yanga
observe
Yellowbrandprotectionbot
other
Yellowbrandprotectionbot
observe
YelpBot
ai assistant
Yelp
YelpBot
observe
Yeti
search engine
Naver
Yeti
allow
Yext
ai assistant
Yext
Yext
observe
Yext Inc
seo crawler
Yext
YextBot
challenge
YGS Group Falconer Scraper
ai crawler
observe
YisouSpider
search engine
Yisou
YisouSpider
allow
YiyanBot
ai assistant
Baidu
YiyanBot
observe
Yokoy Group Webhooks
other
Yokoy Group AG
allow
yoozBot
other
yoozBot
observe
YouBot
search engine
You.com
YouBot
allow
YoudaoBot
other
YoudaoBot
observe
Youmag
other
Youmag
observe
Zabbix
other
Zabbix
observe
ZanistaBot
ai crawler
Zanista
ZanistaBot
observe
Zao
other
Zao
observe