Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
2,114 results
WebTrackrCrawler
ai assistant
Affsignal
WebTrackrCrawler
observe
webzio
ai assistant
Webz.io
webzio
observe
webzio-extended
ai scraper
Webz.io
webzio-extended
observe
WebZIP
other
WebZIP
observe
WellKnownBot
ai coding agent
Well-Known
WellKnownBot
observe
WepchSearchEngine
search engine
Wepch
WepchSearchEngine
allow
What10Bot
search engine
What10Bot
allow
WhatsApp
ai assistant
Meta
WhatsApp
observe
WhiteCircleTrustBot
ai assistant
White Circle
WhiteCircleTrustBot
observe
Who.is Bot
other
observe
WhyNoIPv6Bot
ai assistant
Why No IPv6
WhyNoIPv6Bot
observe
WikiDo
other
WikiDo
observe
WincherBot
other
WincherBot
observe
winello
other
winello
observe
WinHTTrack
other
WinHTTrack
observe
WireReaderBot
ai assistant
Wire
WireReaderBot
observe
WiseGuys Robot
other
WiseGuys
observe
wknd-bot
ai assistant
Wunderkind
wknd-bot
observe
WkndHealthCheckBot
monitoring
Wunderkind
wknd-bot
allow
WMF Citoid
other
Wikimedia Foundation
observe
WMF Zotero Translation Server
other
Wikimedia Foundation
observe
wocbot
other
wocbot
observe
woobot
other
woobot
observe
woorankreview
seo crawler
WooRank
woorankreview
challenge
WordCountBot
social media
Weglot
WordCountBot
allow
Wordfence Central
security
Wordfence
allow
Wordpress Namecheap
other
Namecheap
observe
WordupInfoSearch
other
WordupInfoSearch
observe
woriobot
other
woriobot
observe
Worldline Bot
other
Worldline
allow
WorldPay
other
WorldPay
allow
WormlyBot
monitoring
Wormly
WormlyBot
allow
wotbox
other
wotbox
observe
WOVN Crawler
other
Wovn Technologies, Inc.
WOVN
allow
WovnCrawler
ai assistant
WOVN
WovnCrawler
observe
WP Time Capsule
other
WP Time Capsule
allow
WP Umbrella
monitoring
WP Umbrella
WPUmbrella
allow
WP-Rocket-SaaS
ai coding agent
WP Rocket
WP-Rocket-SaaS
observe
WP-Safe-Scanner
other
WP-Safe-Scanner
observe
WP-VerProbe
other
WP-VerProbe
observe
WP.com mShots
ai assistant
Automattic
observe
wpbot
other
wpbot
observe
WPMU DEV Broken Link Checker
monitoring
WPMUDEV
WPMU
allow
WPMU DEV Broken Link Checker Local Engine
other
observe
WPMU DEV Broken Link Checker Spider
ai coding agent
WPMU DEV
WPMU
observe
WPMU DEV Hub
ai coding agent
WPMU DEV
observe
wpmudev
ai coding agent
WPMUDEV
WordPress
observe
WPMUDEV Uptime Monitor
ai coding agent
WPMU DEV
observe
WPMUDEV Uptime Monitor 5.0
monitoring
WPMUDEV
WPMUDEV Uptime Monitor 5.0 (https:
allow
WPScan
security
WPScan LLC
WPScan
challenge
WRTNBot
ai agent
Wrtn
WRTNBot
observe
ws-bot-v1
other
ws-bot-v1
observe
WSM
other
WSM
observe
WTotem
security
WebTotem
WTotem
allow
WTotemBot
security
WebTotem
WTotemBot
allow
WWW-Collector-E
other
WWW-Collector-E
observe
WWW-Mechanize
other
WWW-Mechanize
observe
www.deadlinkchecker.com
other
www.deadlinkchecker.com
observe
www.uptime.com
other
www.uptime.com
observe
xAI-Grok
ai agent
xAI
xAI-Grok
observe