Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
908 results · Other
Website Quester
other
Website
observe
Website-info.net-Robot
other
Website-info.net-Robot
observe
WebsiteMetadataRetriever
other
WebsiteMetadataRetriever
observe
WebStripper
other
WebStripper
observe
webtext-bot
other
webtext-bot
observe
WebZIP
other
WebZIP
observe
Who.is Bot
other
observe
WikiDo
other
WikiDo
observe
WincherBot
other
WincherBot
observe
winello
other
winello
observe
WinHTTrack
other
WinHTTrack
observe
WiseGuys Robot
other
WiseGuys
observe
WMF Citoid
other
Wikimedia Foundation
observe
WMF Zotero Translation Server
other
Wikimedia Foundation
observe
wocbot
other
wocbot
observe
woobot
other
woobot
observe
Wordpress Namecheap
other
Namecheap
observe
WordupInfoSearch
other
WordupInfoSearch
observe
woriobot
other
woriobot
observe
Worldline Bot
other
Worldline
allow
WorldPay
other
WorldPay
allow
wotbox
other
wotbox
observe
WOVN Crawler
other
Wovn Technologies, Inc.
WOVN
allow
WP Time Capsule
other
WP Time Capsule
allow
WP-Safe-Scanner
other
WP-Safe-Scanner
observe
WP-VerProbe
other
WP-VerProbe
observe
wpbot
other
wpbot
observe
WPMU DEV Broken Link Checker Local Engine
other
observe
ws-bot-v1
other
ws-bot-v1
observe
WSM
other
WSM
observe
WWW-Collector-E
other
WWW-Collector-E
observe
WWW-Mechanize
other
WWW-Mechanize
observe
www.deadlinkchecker.com
other
www.deadlinkchecker.com
observe
www.uptime.com
other
www.uptime.com
observe
Xbot
other
Xbot
observe
xenU
other
xenU
observe
Xenu Link Sleuth
other
observe
Xenu's
other
Xenu's
observe
Xenu's Link Sleuth 1.1c
other
Xenu&
observe
Xing Bot
other
Xing
observe
XML Sitemaps Generator
other
observe
xovibot
other
xovibot
observe
Xpanse-bot
other
Xpanse-bot
observe
XY Archive Compliance Bot
other
XY Archive Compliance
XY-Archive-Compliance
observe
Y!J-ASR
other
Y!J-ASR
observe
Y!J-BRW
other
Y!J-BRW
observe
Y!J-DLC
other
Y!J-DLC
observe
Yahoo Pipes 1.0
other
observe
YandexVideo
other
YandexVideo
observe
YandoriRSSBot
other
YandoriRSSBot
observe
yanga
other
yanga
observe
Yellowbrandprotectionbot
other
Yellowbrandprotectionbot
observe
Yokoy Group Webhooks
other
Yokoy Group AG
allow
yoozBot
other
yoozBot
observe
YoudaoBot
other
YoudaoBot
observe
Youmag
other
Youmag
observe
Zabbix
other
Zabbix
observe
Zao
other
Zao
observe
Zapier
other
Zapier Inc.
Zapier
allow
Zealbot
other
Zealbot
observe