Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
497 results
Svix Inc.
sender-9YMgn
Swfiteq Ltd
Swifteq
DataDog
Synthetic
Telegram
TelegramBot
Ask
Teoma
Termly
TermlyBot
Ceramic
TerraCotta
Thousand Eyes
Thousand
TinEye
TinEye
TinEye
TinEye-bot
TinEye
TinEye-bot-live
TinEye
TinEye-Web
Bytedance
zhanzhang.toutiao.com
Trendiction S.A.
trendictionbot
Trustly Group AB
Automattic
Tumblr
Turnitin
Turnitin (https:
Twilio, Inc.
Splunk
TwinWaveScanner
Twitterbot
Updown
UpDownBot
Uptime
Uptime
GoDaddy
UptimeMonitoring
UptimeBot
UptimeBot
Uptimerobot
UptimeRobot
Uptimia
Uptimia
Uptrends GmbH
uptrends
Usercentrics
Usercentricsbot
Automattic
VaultPress
Project Honey Pot
Verispider
VirusTotal
VirusTotalScan
visionheight.com/scan
Evil Source Platform
VivianCrawler
WatchBot
Watchbot
Watchful LLC
Watchful
WebPageTest
webpagetest
WebsitePulse
WebsitePulseBot
WebStatus247
Webstatus247
WebTotem
WTotem
Wepch
WepchSearchEngine
What10Bot
Wunderkind
wknd-bot
Weglot
WordCountBot
Wordfence
Worldline
WorldPay
Wormly
WormlyBot
Wovn Technologies, Inc.
WOVN
WP Time Capsule
WP Umbrella
WPUmbrella
WPMUDEV
WPMU
WPMUDEV
WPMUDEV Uptime Monitor 5.0 (https:
WebTotem
WTotem
WebTotem
WTotemBot
XY Planning Network
XY-Archive-Compliance-Archiver
Yahoo
YaCy
yacybot
Yahoo
Yahoo Link Preview
Yahoo
YahooMailProxy
Yahoo