Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,426 results
WebCopier v3.2a
other
observe
WebCrawler
other
WebCrawler
observe
WebDataStats
other
WebDataStats
observe
WebEnhancer
other
WebEnhancer
observe
WebmasterWorldForumBot
other
WebmasterWorldForumBot
observe
webmon
other
webmon
observe
weborama-fetcher
ai assistant
Weborama
weborama-fetcher
observe
WebReaper
other
WebReaper
observe
WebSauger
other
WebSauger
observe
WebScraperBot
other
WebScraperBot
observe
websearch-bot
other
websearch-bot
observe
Website Quester
other
Website
observe
Website-info.net-Robot
other
Website-info.net-Robot
observe
WebsiteMetadataRetriever
other
WebsiteMetadataRetriever
observe
WebsiteOps
ai coding agent
Watchful
WebsiteOps
observe
websitepulse checker
ai coding agent
WebSitePulse
websitepulse checker
observe
WebStripper
other
WebStripper
observe
webtext-bot
other
webtext-bot
observe
WebTrackrCrawler
ai assistant
Affsignal
WebTrackrCrawler
observe
webzio
ai assistant
Webz.io
webzio
observe
webzio-extended
ai scraper
Webz.io
webzio-extended
observe
WebZIP
other
WebZIP
observe
WellKnownBot
ai coding agent
Well-Known
WellKnownBot
observe
WhatsApp
ai assistant
Meta
WhatsApp
observe
WhiteCircleTrustBot
ai assistant
White Circle
WhiteCircleTrustBot
observe
Who.is Bot
other
observe
WhyNoIPv6Bot
ai assistant
Why No IPv6
WhyNoIPv6Bot
observe
WikiDo
other
WikiDo
observe
WincherBot
other
WincherBot
observe
winello
other
winello
observe
WinHTTrack
other
WinHTTrack
observe
WireReaderBot
ai assistant
Wire
WireReaderBot
observe
WiseGuys Robot
other
WiseGuys
observe
wknd-bot
ai assistant
Wunderkind
wknd-bot
observe
WMF Citoid
other
Wikimedia Foundation
observe
WMF Zotero Translation Server
other
Wikimedia Foundation
observe
wocbot
other
wocbot
observe
woobot
other
woobot
observe
Wordpress Namecheap
other
Namecheap
observe
WordupInfoSearch
other
WordupInfoSearch
observe
woriobot
other
woriobot
observe
wotbox
other
wotbox
observe
WovnCrawler
ai assistant
WOVN
WovnCrawler
observe
WP-Rocket-SaaS
ai coding agent
WP Rocket
WP-Rocket-SaaS
observe
WP-Safe-Scanner
other
WP-Safe-Scanner
observe
WP-VerProbe
other
WP-VerProbe
observe
WP.com mShots
ai assistant
Automattic
observe
wpbot
other
wpbot
observe
WPMU DEV Broken Link Checker Local Engine
other
observe
WPMU DEV Broken Link Checker Spider
ai coding agent
WPMU DEV
WPMU
observe
WPMU DEV Hub
ai coding agent
WPMU DEV
observe
wpmudev
ai coding agent
WPMUDEV
WordPress
observe
WPMUDEV Uptime Monitor
ai coding agent
WPMU DEV
observe
WRTNBot
ai agent
Wrtn
WRTNBot
observe
ws-bot-v1
other
ws-bot-v1
observe
WSM
other
WSM
observe
WWW-Collector-E
other
WWW-Collector-E
observe
WWW-Mechanize
other
WWW-Mechanize
observe
www.deadlinkchecker.com
other
www.deadlinkchecker.com
observe
www.uptime.com
other
www.uptime.com
observe