Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,953 results
WPMU DEV Broken Link Checker
monitoring
WPMUDEV
WPMU
allow
WPMU DEV Broken Link Checker Local Engine
other
observe
WPMU DEV Broken Link Checker Spider
ai coding agent
WPMU DEV
WPMU
observe
WPMU DEV Hub
ai coding agent
WPMU DEV
observe
wpmudev
ai coding agent
WPMUDEV
WordPress
observe
WPMUDEV Uptime Monitor
ai coding agent
WPMU DEV
observe
WPMUDEV Uptime Monitor 5.0
monitoring
WPMUDEV
WPMUDEV Uptime Monitor 5.0 (https:
allow
WPScan
security
WPScan LLC
WPScan
challenge
WPUmbrella
ai coding agent
WP Umbrella
WPUmbrella
observe
WRTNBot
ai agent
Wrtn
WRTNBot
observe
ws-bot-v1
other
ws-bot-v1
observe
WSM
other
WSM
observe
WTotem
security
WebTotem
WTotem
allow
WTotemBot
security
WebTotem
WTotemBot
allow
WWW-Collector-E
other
WWW-Collector-E
observe
WWW-Mechanize
other
WWW-Mechanize
observe
www.deadlinkchecker.com
other
www.deadlinkchecker.com
observe
www.uptime.com
other
www.uptime.com
observe
xAI-SearchBot
ai crawler
xAI
xAI-SearchBot
observe
Xbot
other
Xbot
observe
xenU
other
xenU
observe
Xenu Link Sleuth
other
observe
Xenu's
other
Xenu's
observe
Xenu's Link Sleuth 1.1c
other
observe
Xing Bot
other
Xing
observe
XML Sitemaps Generator
other
observe
xovibot
other
xovibot
observe
XoviOnpageCrawler
seo crawler
XOVI
XoviOnpageCrawler
challenge
Xpanse-bot
other
Xpanse-bot
observe
XY Archive Compliance Bot
other
XY Archive Compliance
XY-Archive-Compliance
observe
XY-Archive-Compliance
archiver
XY Archive Compliance
XY-Archive-Compliance
allow
XY-Archive-Compliance-Archiver
archiver
XY Planning Network
XY-Archive-Compliance-Archiver
allow
Y!J-ASR
other
Y!J-ASR
observe
Y!J-BRW
other
Y!J-BRW
observe
Y!J-BRZ/YATSHA crawler
search engine
Yahoo
allow
Y!J-DLC
other
Y!J-DLC
observe
yacybot
search engine
YaCy
yacybot
allow
YaDirectFetcher
ai assistant
Yandex
YaDirectFetcher
observe
Yahoo Ad Monitoring
seo crawler
Yahoo
challenge
Yahoo Japan
seo crawler
Yahoo Japan Corporation
Yahoo
challenge
Yahoo Japan SEO Crawler
seo crawler
Yahoo
J-BRW
challenge
Yahoo Link Preview
social media
Yahoo
Yahoo Link Preview
allow
Yahoo Mail
social media
Yahoo
YahooMailProxy
allow
Yahoo Pipes 1.0
other
observe
Yahoo Slurp
search engine
Yahoo
allow
Yahoo! Slurp
search engine
Yahoo
allow
YahooCacheSystem
social media
Yahoo
YahooCacheSystem
allow
YahooMailProxy
social media
Yahoo
YahooMailProxy
allow
YaK
ai assistant
Linkfluence
YaK
observe
YandexAdditional
ai crawler
Yandex
YandexAdditional
observe
YandexAdditionalBot
ai crawler
Yandex
YandexAdditionalBot
observe
YandexBot
search engine
Yandex
http://yandex.com/bots
allow
rDNS
YandexImages
search engine
Yandex
YandexImages
allow
rDNS
YandexMobileBot
search engine
Yandex
YandexMobileBot
allow
rDNS
YandexRenderResourcesBot
search engine
Yandex
YandexRenderResourcesBot
allow
rDNS
YandexVertis
search engine
Yandex
YandexVertis
allow
rDNS
YandexVideo
other
YandexVideo
observe
YandoriRSSBot
other
YandoriRSSBot
observe
yanga
other
yanga
observe
Yellowbrandprotectionbot
other
Yellowbrandprotectionbot
observe