Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
2,114 results
dcrawl
DLC Websites
deadlinkchecker
DeckAgent
Lumar
deepcrawl
Lumar
deepcrawl.com
deepnoc
DeepSeek
DeepSeekBot
Desearch
DesearchBot
Detectify
Detectify
DeuSu
Devin AI
Devin
Diffbot
Diffbot
Digg
DigiCert
Digicert
Digimind
Digincore
Direqt
Anomura
discobot
Discord, Inc.
Discordbot
Disqus
DittoSpyder
Dr. Link Check
DLC
dlvr.it
dlvr.it
dlvr.it
Dmbot
Dotcom-Monitor
DMBrowser
DnBCrawler-Analytics
DNSCrawler
DNSScanner
DnyzBot
Electronic Frontier Foundation
DocSpot
DocSpotBot
Dotcom-Monitor
DMBrowser
ByteDance
Dola-SearchBot
Domain
DomainCrawler
DRM3
DomainDrift
DomainEvidenceBot
DomainMapper
Domains
Pragmatic Security
DomainScores
Game Lounge
DomainStatsBot
DominicBot
OpenAI
DotSee
DotSee-HealthCheck
ByteDance
Doubaobot
Download
UpDownSideways
DowntimeDetector
Dr. Link Check
Dragon Metrics
Dragonbot
Publisher Discovery
Dragonfly
Drata
Dratabot
Drata
Dratabot
DreamHost
DreamHost
Drift Security
drift-agent
drupact
Drupalbot
Copyright Clearance Center
ds9
DubBot
Dubbotbot