Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
844 results · Other
Website Quester
other
Website
observe
Website-info.net-Robot
other
Website-info.net-Robot
observe
WebStripper
other
WebStripper
observe
webtext-bot
other
webtext-bot
observe
WebZIP
other
WebZIP
observe
WikiDo
other
WikiDo
observe
WincherBot
other
WincherBot
observe
winello
other
winello
observe
WinHTTrack
other
WinHTTrack
observe
WiseGuys Robot
other
WiseGuys
observe
WMF Citoid
other
Wikimedia Foundation
observe
WMF Zotero Translation Server
other
Wikimedia Foundation
observe
wocbot
other
wocbot
observe
woobot
other
woobot
observe
Wordpress Namecheap
other
Namecheap
observe
WordupInfoSearch
other
WordupInfoSearch
observe
woriobot
other
woriobot
observe
Worldline Bot
other
Worldline
allow
WorldPay
other
WorldPay
allow
wotbox
other
wotbox
observe
WOVN Crawler
other
Wovn Technologies, Inc.
WOVN
allow
WP Time Capsule
other
WP Time Capsule
allow
wpbot
other
wpbot
observe
WPMU DEV Broken Link Checker Local Engine
other
observe
ws-bot-v1
other
ws-bot-v1
observe
WSM
other
WSM
observe
WWW-Collector-E
other
WWW-Collector-E
observe
WWW-Mechanize
other
WWW-Mechanize
observe
www.deadlinkchecker.com
other
www.deadlinkchecker.com
observe
www.uptime.com
other
www.uptime.com
observe
Xbot
other
Xbot
observe
xenU
other
xenU
observe
Xenu Link Sleuth
other
observe
Xenu's
other
Xenu's
observe
Xenu's Link Sleuth 1.1c
other
observe
Xing Bot
other
Xing
observe
XML Sitemaps Generator
other
observe
xovibot
other
xovibot
observe
Xpanse-bot
other
Xpanse-bot
observe
XY Archive Compliance Bot
other
XY Archive Compliance
XY-Archive-Compliance
observe
Y!J-ASR
other
Y!J-ASR
observe
Y!J-BRW
other
Y!J-BRW
observe
Y!J-DLC
other
Y!J-DLC
observe
Yahoo Pipes 1.0
other
observe
YandexVideo
other
YandexVideo
observe
YandoriRSSBot
other
YandoriRSSBot
observe
yanga
other
yanga
observe
Yellowbrandprotectionbot
other
Yellowbrandprotectionbot
observe
Yokoy Group Webhooks
other
Yokoy Group AG
allow
yoozBot
other
yoozBot
observe
YoudaoBot
other
YoudaoBot
observe
Youmag
other
Youmag
observe
Zabbix
other
Zabbix
observe
Zao
other
Zao
observe
Zapier
other
Zapier Inc.
Zapier
allow
Zealbot
other
Zealbot
observe
zenback bot
other
zenback
observe
Zendesk Webhook
other
Zendesk
allow
zeus
other
zeus
observe
Zeus Link Scout
other
observe