Updated daily
Crawler Catalog
1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
1,876 results
webmon
other
webmon
observe
weborama-fetcher
ai assistant
Weborama
weborama-fetcher
observe
webpagetest
monitoring
WebPageTest
webpagetest
allow
WebReaper
other
WebReaper
observe
WebSauger
other
WebSauger
observe
WebScraperBot
other
WebScraperBot
observe
websearch-bot
other
websearch-bot
observe
Website
other
Website
observe
Website-info.net-Robot
other
Website-info.net-Robot
observe
WebsiteOps
ai coding agent
Watchful
WebsiteOps
observe
websitepulse
ai coding agent
WebSitePulse
websitepulse
observe
WebsitePulseBot
monitoring
WebsitePulse
WebsitePulseBot
allow
WebSpiderMount
seo crawler
AspenTechLabs Inc
webspidermount
challenge
Webstatus247
monitoring
WebStatus247
allow
WebStripper
other
WebStripper
observe
webtext-bot
other
webtext-bot
observe
WebTotemBot
security
WebTotem
allow
WebTrackrCrawler
ai assistant
Affsignal
WebTrackrCrawler
observe
webzio
ai assistant
Webz.io
webzio
observe
webzio-extended
ai scraper
Webz.io
webzio-extended
observe
WebZIP
other
WebZIP
observe
WellKnownBot
ai coding agent
Well-Known
WellKnownBot
observe
WepchSearchEngine
search engine
Wepch
WepchSearchEngine
allow
WhatsApp
ai assistant
Meta
WhatsApp
observe
WikiDo
other
WikiDo
observe
WincherBot
other
WincherBot
observe
winello
other
winello
observe
WinHTTrack
other
WinHTTrack
observe
WireReaderBot
ai assistant
Wire
WireReaderBot
observe
WiseGuys
other
WiseGuys
observe
wknd-bot
ai assistant
Wunderkind
wknd-bot
observe
WkndHealthCheckBot
monitoring
Wunderkind
allow
WMF
other
Wikimedia
WMF
observe
WMF Citoid
other
Wikimedia Foundation
observe
WMF Zotero Translation Server
other
Wikimedia Foundation
observe
wocbot
other
wocbot
observe
woobot
other
woobot
observe
woorankreview
seo crawler
WooRank
woorankreview
challenge
WordCountBot
social media
Weglot
WordCountBot
allow
Wordfence Central
security
Wordfence
allow
Wordpress Namecheap
other
Namecheap
observe
WordupInfoSearch
other
WordupInfoSearch
observe
woriobot
other
woriobot
observe
Worldline Bot
other
Worldline
allow
WorldPay
other
WorldPay
allow
WormlyBot
monitoring
Wormly
WormlyBot
allow
wotbox
other
wotbox
observe
WOVN Crawler
other
Wovn Technologies, Inc.
WOVN
allow
WovnCrawler
ai assistant
WOVN
WovnCrawler
observe
WP Time Capsule
other
WP Time Capsule
allow
WP Umbrella
monitoring
WP Umbrella
WPUmbrella
allow
WP-Rocket-SaaS
ai coding agent
WP Rocket
WP-Rocket-SaaS
observe
wpbot
other
wpbot
observe
WPMU
ai coding agent
WPMU DEV
WPMU
observe
WPMU DEV Broken Link Checker
monitoring
WPMUDEV
WPMU
allow
wpmudev
ai coding agent
WPMUDEV
wpmudev
observe
WPMUDEV Uptime Monitor 5.0
monitoring
WPMUDEV
WPMUDEV
allow
WPScan
security
WPScan LLC
challenge
WRTNBot
ai agent
Wrtn
WRTNBot
observe
ws-bot-v1
other
ws-bot-v1
observe