Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
75 results · AI Coding Agent
2IP
Cloudflare
AgentReadinessScanner
Stack Moxie
AutomatonHeadlessChrome
Bugsnag
Binary Canary
Chirp
Anthropic
Claude-Code
GitHub
Code
CrazyEgg
Critical CSS
Criticalcss.com
xAI
Cursor
Datadog
Datadog Agent
Datadog
Dr. Link Check
DLC
Dotcom-Monitor
DMBrowser
Electronic Frontier Foundation
DreamHost
DreamHost
DubBot
Dubbotbot
Electronic Frontier
Amazon
ELB-HealthChecker
Freshworks
FreshpingBot
Microsoft / GitHub
Google-Gemini-CLI
Google-Structured-Data-Testing-Tool
GoogleAssociationService
Grafana Labs
grafana
Honeybadger
Honeybadger
Automattic
jetmon
Kinsta
kinsta
LogicMonitor
360 Monitoring
monitoring360
360 Monitoring
monitoring360bot
Montastic
montastic-monitor
New Relic
NewRelicbot
NitroPack
Nitro-Optimizer-Agent
NixStats
NIXStatsbot
Oh Dear
OhDear.app
OnlineOrNot
OnlineOrNot.com_bot
Anomaly
opencode
Parse.ly
parsely.com
SolarWinds
Pingdom.com_bot
PingPing
pingping.io
Paessler
PRTGCloudBot
Pulsetic
pulsetic
Pulsetic
pulsetic.com
Microsoft
PWABuilderHttpAgent
Rackspace
Scrunch
Scrunchbot
Semrush
SemrushBot-SWA
Sentry
SentryUptimeBot
Nexigen Digital
StackPath
Silktide
Silktide
Stape
Stape
Swifteq
SwifteqLinkChecker
Grafana Labs
synthetic-monitoring-agent
T3Versions
t3versionsBot
Testomato
Testomatobot
ByteDance
Trae
Updown
updown.io
updown.io