Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,284 results
webspidermount
ai assistant
AspenTechLabs
webspidermount
observe
WebStripper
other
WebStripper
observe
webtext-bot
other
webtext-bot
observe
WebTrackrCrawler
ai assistant
Affsignal
WebTrackrCrawler
observe
webzio
ai assistant
Webz.io
webzio
observe
webzio-extended
ai scraper
Webz.io
webzio-extended
observe
WebZIP
other
WebZIP
observe
WellKnownBot
ai coding agent
Well-Known
WellKnownBot
observe
WhatsApp
ai assistant
Meta
WhatsApp
observe
WikiDo
other
WikiDo
observe
WincherBot
other
WincherBot
observe
winello
other
winello
observe
WinHTTrack
other
WinHTTrack
observe
WireReaderBot
ai assistant
Wire
WireReaderBot
observe
WiseGuys Robot
other
WiseGuys
observe
wknd-bot
ai assistant
Wunderkind
wknd-bot
observe
WMF Citoid
other
Wikimedia Foundation
observe
WMF Zotero Translation Server
other
Wikimedia Foundation
observe
wocbot
other
wocbot
observe
woobot
other
woobot
observe
Wordpress Namecheap
other
Namecheap
observe
WordupInfoSearch
other
WordupInfoSearch
observe
woriobot
other
woriobot
observe
wotbox
other
wotbox
observe
WovnCrawler
ai assistant
WOVN
WovnCrawler
observe
WP-Rocket-SaaS
ai coding agent
WP Rocket
WP-Rocket-SaaS
observe
wpbot
other
wpbot
observe
WPMU DEV Broken Link Checker Local Engine
other
observe
WPMU DEV Broken Link Checker Spider
ai coding agent
WPMU DEV
WPMU
observe
WPMU DEV Hub
ai coding agent
WPMU DEV
observe
wpmudev
ai coding agent
WPMUDEV
WordPress
observe
WPMUDEV Uptime Monitor
ai coding agent
WPMU DEV
observe
WPUmbrella
ai coding agent
WP Umbrella
WPUmbrella
observe
WRTNBot
ai agent
Wrtn
WRTNBot
observe
ws-bot-v1
other
ws-bot-v1
observe
WSM
other
WSM
observe
WWW-Collector-E
other
WWW-Collector-E
observe
WWW-Mechanize
other
WWW-Mechanize
observe
www.deadlinkchecker.com
other
www.deadlinkchecker.com
observe
www.uptime.com
other
www.uptime.com
observe
xAI-SearchBot
ai crawler
xAI
xAI-SearchBot
observe
Xbot
other
Xbot
observe
xenU
other
xenU
observe
Xenu Link Sleuth
other
observe
Xenu's
other
Xenu's
observe
Xenu's Link Sleuth 1.1c
other
observe
Xing Bot
other
Xing
observe
XML Sitemaps Generator
other
observe
xovibot
other
xovibot
observe
Xpanse-bot
other
Xpanse-bot
observe
XY Archive Compliance Bot
other
XY Archive Compliance
XY-Archive-Compliance
observe
Y!J-ASR
other
Y!J-ASR
observe
Y!J-BRW
other
Y!J-BRW
observe
Y!J-DLC
other
Y!J-DLC
observe
YaDirectFetcher
ai assistant
Yandex
YaDirectFetcher
observe
Yahoo Pipes 1.0
other
observe
YaK
ai assistant
Linkfluence
YaK
observe
YandexAdditional
ai crawler
Yandex
YandexAdditional
observe
YandexAdditionalBot
ai crawler
Yandex
YandexAdditionalBot
observe
YandexVideo
other
YandexVideo
observe