Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
122 results · Monitoring
360Monitoring
360monitoring
AccessStatus
AccessStatus
ActiveComply
ActiveComply Bot
Adagio
Adagiobot
Advailo
AgencyToolsPro
AgencyToolsPro
Smartbear
AlertSite
Area360
A360-Search
Audisto
Audisto
Automaton
Automaton
Better Stack
Better
BigScoots
BigScootsMonitor
Binary Canary
binarycanary
BlogVault
BlogVault
Brynson
BrynsonComplianceBot
Bugs Nag
BugsNag
BuiltWith
BuiltWith
Catchpoint
Catchpoint
Checkly
Checkly
CheckView.io
CheckView
Cloudflare
Cloudflare
CloudflareDiagnostics
Cloudflare
curly-
Cloudtrellis
Cloudtrellis
Cludo
cludo.com
ContentKing
contentking
Contentsquare
ContentSquare
CrazyEgg
Crazy
DLC Websites
deadlinkchecker
Lumar
deepcrawl
Dotcom-Monitor
DMBrowser
Dr. Link Check
EasyDNS
easyDNS
elmah.io
elmahio-uptimebot
Sebastian Foth - Software Solutions
eRepublik.tools
Evo Agency Ltd.
EvoUptimeBot
FTW Entertainment LLC
FDL
Foregenix Limited
Foregenix
Full Story
FullStoryBot
Gatus
Gatus
Ghost Inspector
Ghost
Google-Structured-Data-Testing
Grafana Labs
synthetic-monitoring-agent
Grafana Labs
GTmetrix
GTmetrix
Guest Posts
GuestpostsBot
HetrixTools Inc
HetrixTools
Honeybadger Industries
HoneybadgerBot
HostTracker
HostTracker
Hotjar
Hotjar
Internet Archive
InternetArchiveBot
IsDownBot
IsDownBot
Jagged Pixel Inc.
www.getuptime.co
Automattic
Jetpack
Kargo
KargoBot-Artemis
Kinsta
kinsta-bot
LinkTiger, Inc.
LinkTiger
LogicMonitor
LogicMonitor InternalService
Macrobondbot
Macrobondbot
Mavifinds
Mavifinds