Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
77 results · AI Coding Agent
2IP
Cloudflare
AgentReadinessScanner
Stack Moxie
AutomatonHeadlessChrome
Bugsnag
Binary Canary
Chirp
Anthropic
Claude-Code
GitHub
Code
OpenAI
Codex
Critical CSS
Criticalcss.com
xAI
Cursor
Datadog
Datadog Agent
Datadog
Dr. Link Check
DLC
Dotcom-Monitor
DMBrowser
Electronic Frontier Foundation
DotSee
DotSee-HealthCheck
UpDownSideways
DowntimeDetector
DreamHost
DreamHost
DubBot
Dubbotbot
Electronic Frontier
Amazon
ELB-HealthChecker
Freshworks
FreshpingBot
Microsoft / GitHub
Google-Gemini-CLI
Google-Structured-Data-Testing-Tool
GoogleAssociationService
Grafana Labs
grafana
Grafana Labs
Grafana&
Honeybadger
Honeybadger
Automattic
jetmon
KamenX
Let's Encrypt
Let&
LogicMonitor
ManageWP
ManageWP-UptimeMonitoring
360 Monitoring
monitoring360
360 Monitoring
monitoring360bot
New Relic
NewRelicbot
NitroPack
Nitro-Optimizer-Agent
NixStats
NIXStatsbot
Oh Dear
OhDear.app
OnlineOrNot
OnlineOrNot.com_bot
Anomaly
opencode
Overture Maps
Overture-Maps-LinkChecker
Parse.ly
parsely.com
SolarWinds
Pingdom.com_bot
PingPing
pingping.io
Paessler
PRTGCloudBot
Pulsetic
pulsetic.com
Microsoft
PWABuilderHttpAgent
Qoder
qodercli
Rackspace
Scrunch
Scrunchbot
Sentry
SentryUptimeBot
Nexigen Digital
StackPath
Silktide
Silktide
Swifteq
SwifteqLinkChecker
Grafana Labs
synthetic-monitoring-agent
T3Versions
t3versionsBot
Testomato
Testomatobot