Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
122 results · Monitoring
Mediaboard
mediaboardbot
Medialogia
MedialogiaBot
Monitage
Monitage
Monitis
monitis
Acquia
Acquia
MonSpark
MonSpark
Montastic
MontasticMonitor
Netumo
netumo
Netvigie
NETVIGIE
New Relic
NewRelicbot
NixStats
NixStatsMonitoringBot
Nodeping
NodePing
Oh Dear
OhDear
OneTrust, LLC.
Onetrust
OnlineOrNot
OnlineOrNot.com_bot_1.0_(https:
Outsell Corporation
OutsellURLValidator
Paessler
PRTGCloudBot
Panopta
Panopta
Parse.ly
parse.ly scraper
Pingdom
Pingdom
PingPing
pingping.io/
Pulsetic
pulsetic.com (+https:
Qualys
Qualys SSL Scanner
Rackspace
Rackspace Monitoring
Niclas Papst
RetroListeCOM
Robin Education
Robinbot
SE Ranking
SE
Sentry Uptime Bot
Sentry
SequelWP
SequelWP
Stackpath
Server
SISTRIX GmbH
Site24x7
site24x7
SiteUpTime
SiteUptime
SolarWinds
SolarWinds Observability
Spectate
Spectate
Splunk
Rigor
Splunk
Splunk Synthetics
Stape Inc
Stape
StatusCake
statuscake
Swfiteq Ltd
Swifteq
DataDog
Synthetic
Termly
TermlyBot
Thousand Eyes
Thousand
Trendiction S.A.
trendictionbot
Updown
UpDownBot
Uptime
Uptime
GoDaddy
UptimeMonitoring
UptimeBot
UptimeBot
Uptimerobot
UptimeRobot
Uptimia
Uptimia
Uptrends GmbH
uptrends
WatchBot
Watchbot
Watchful LLC
Watchful
WebPageTest
webpagetest
WebsitePulse
WebsitePulseBot
WebStatus247
Webstatus247
Wunderkind
wknd-bot
Wormly
WormlyBot
WP Umbrella
WPUmbrella
WPMUDEV
WPMU