Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
198 results
Statabot
seo crawler
StataCorp LLC
Statabot
challenge
StatsDroneBot
seo crawler
StatsDrone
StatsDroneBot
challenge
StatusNestBacklinkSpider
seo crawler
StatusNest
StatusNestBacklinkSpider
challenge
Taboola
seo crawler
Taboola
Taboolabot
challenge
Trellis-Services
seo crawler
Mediavine
Trellis-Services
challenge
TTD Content
seo crawler
The Trade Desk
TTD-Content
challenge
upday
seo crawler
upday GmbH & Co. KG
upday
challenge
videootv Bot
seo crawler
Digital Green
videootvBot
challenge
W3 Validator Services
seo crawler
World Wide Web Consortium (W3C)
W3
challenge
WebSpiderMount
seo crawler
AspenTechLabs Inc
webspidermount
challenge
woorankreview
seo crawler
WooRank
woorankreview
challenge
WPScan
security
WPScan LLC
WPScan
challenge
XoviOnpageCrawler
seo crawler
XOVI
XoviOnpageCrawler
challenge
Yahoo Ad Monitoring
seo crawler
Yahoo
challenge
Yahoo Japan
seo crawler
Yahoo Japan Corporation
Yahoo
challenge
Yahoo Japan SEO Crawler
seo crawler
Yahoo
J-BRW
challenge
Yext Inc
seo crawler
Yext
YextBot
challenge
Zoombot
seo crawler
SEOZoom
Zoombot
challenge