Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,953 results
WebCopier
other
WebCopier
observe
WebCopier v.2.2
other
observe
WebCopier v3.2a
other
observe
WebCrawler
other
WebCrawler
observe
WebDataStats
other
WebDataStats
observe
WebEnhancer
other
WebEnhancer
observe
WebmasterWorldForumBot
other
WebmasterWorldForumBot
observe
webmon
other
webmon
observe
weborama-fetcher
ai assistant
Weborama
weborama-fetcher
observe
webpagetest
monitoring
WebPageTest
webpagetest
allow
WebReaper
other
WebReaper
observe
WebSauger
other
WebSauger
observe
WebScraperBot
other
WebScraperBot
observe
websearch-bot
other
websearch-bot
observe
Website Quester
other
Website
observe
Website-info.net-Robot
other
Website-info.net-Robot
observe
WebsiteOps
ai coding agent
Watchful
WebsiteOps
observe
websitepulse checker
ai coding agent
WebSitePulse
websitepulse checker
observe
WebsitePulseBot
monitoring
WebsitePulse
WebsitePulseBot
allow
WebSpiderMount
seo crawler
AspenTechLabs Inc
webspidermount
challenge
webspidermount
ai assistant
AspenTechLabs
webspidermount
observe
Webstatus247
monitoring
WebStatus247
Webstatus247
allow
WebStripper
other
WebStripper
observe
webtext-bot
other
webtext-bot
observe
WebTotemBot
security
WebTotem
WTotem
allow
WebTrackrCrawler
ai assistant
Affsignal
WebTrackrCrawler
observe
webzio
ai assistant
Webz.io
webzio
observe
webzio-extended
ai scraper
Webz.io
webzio-extended
observe
WebZIP
other
WebZIP
observe
WellKnownBot
ai coding agent
Well-Known
WellKnownBot
observe
WepchSearchEngine
search engine
Wepch
WepchSearchEngine
allow
WhatsApp
ai assistant
Meta
WhatsApp
observe
WikiDo
other
WikiDo
observe
WincherBot
other
WincherBot
observe
winello
other
winello
observe
WinHTTrack
other
WinHTTrack
observe
WireReaderBot
ai assistant
Wire
WireReaderBot
observe
WiseGuys Robot
other
WiseGuys
observe
wknd-bot
ai assistant
Wunderkind
wknd-bot
observe
WkndHealthCheckBot
monitoring
Wunderkind
wknd-bot
allow
WMF Citoid
other
Wikimedia Foundation
observe
WMF Zotero Translation Server
other
Wikimedia Foundation
observe
wocbot
other
wocbot
observe
woobot
other
woobot
observe
woorankreview
seo crawler
WooRank
woorankreview
challenge
WordCountBot
social media
Weglot
WordCountBot
allow
Wordfence Central
security
Wordfence
allow
Wordpress Namecheap
other
Namecheap
observe
WordupInfoSearch
other
WordupInfoSearch
observe
woriobot
other
woriobot
observe
Worldline Bot
other
Worldline
allow
WorldPay
other
WorldPay
allow
WormlyBot
monitoring
Wormly
WormlyBot
allow
wotbox
other
wotbox
observe
WOVN Crawler
other
Wovn Technologies, Inc.
WOVN
allow
WovnCrawler
ai assistant
WOVN
WovnCrawler
observe
WP Time Capsule
other
WP Time Capsule
allow
WP Umbrella
monitoring
WP Umbrella
WPUmbrella
allow
WP-Rocket-SaaS
ai coding agent
WP Rocket
WP-Rocket-SaaS
observe
wpbot
other
wpbot
observe