Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,284 results
Datadog
ai coding agent
Datadog
Datadog Agent
observe
Datadog Synthetic
ai coding agent
Datadog
observe
Datafeedwatch
other
Datafeedwatch
observe
datagnionbot
other
datagnionbot
observe
Datanyze
other
Datanyze
observe
Datenbank Crawler
ai scraper
netEstate
Datenbank
observe
Daumoa
other
Daumoa
observe
Dazzle BlueSky Bot
other
Dazzle
observe
dbot
other
dbot
observe
Deck Agent
ai assistant
DeckAgent
observe
signed
deepnoc
other
deepnoc
observe
DeepSeekBot
ai crawler
DeepSeek
DeepSeekBot
observe
DeuSu
other
DeuSu
observe
Devin
ai assistant
Devin AI
Devin
observe
Diffbot
ai data provider
Diffbot
Diffbot
observe
Digg Deeper
other
Digg
observe
Digimind
other
Digimind
observe
Digincore bot
other
Digincore
observe
Direqt Anomura
ai crawler
Direqt
Anomura
observe
discobot
other
discobot
observe
Disqus
other
Disqus
observe
DittoSpyder
other
DittoSpyder
observe
DLC
ai coding agent
Dr. Link Check
DLC
observe
dlvr.it
ai assistant
dlvr.it
dlvr.it
observe
DMBrowser
ai coding agent
Dotcom-Monitor
DMBrowser
observe
DnBCrawler-Analytics
other
DnBCrawler-Analytics
observe
DNSCrawler
other
DNSCrawler
observe
DNSScanner
other
DNSScanner
observe
DnyzBot
other
DnyzBot
observe
Do Not Track Verifier
ai coding agent
Electronic Frontier Foundation
observe
Domain Re-Animator Bot
other
Domain
observe
DomainCrawler
other
DomainCrawler
observe
DomainMapper
other
DomainMapper
observe
Domains Project
other
Domains
observe
DominicBot
other
DominicBot
observe
Doubaobot
ai scraper
ByteDance
Doubaobot
observe
Dow Jones Searchbot
other
observe
Download Ninja
other
Download
observe
Dragonfly
ai assistant
Publisher Discovery
Dragonfly
observe
DreamHost Data Team
ai coding agent
DreamHost
DreamHost
observe
Drift Agent
ai assistant
Drift Security
drift-agent
observe
signed
drupact
other
drupact
observe
Drupalbot
other
Drupalbot
observe
ds9
ai assistant
Copyright Clearance Center
ds9
observe
Dubbotbot
ai coding agent
DubBot
Dubbotbot
observe
DuckAssistbot
ai assistant
DuckDuckGo
DuckAssistBot
observe
DVbot
ai assistant
DoubleVerify
DVbot
observe
e.ventures Investment Crawler
other
e.ventures
observe
EasyBib AutoCite
other
EasyBib
observe
EasyBill Import Manager
other
easybill.de
easybill-ImportManager
observe
easybill-ImportManager
ai assistant
easybill
easybill-ImportManager
observe
easyDNS Monitoring
other
observe
EasyScan
ai assistant
codire GmbH
EasyScan
observe
signed
ec2linkfinder
other
ec2linkfinder
observe
Echobot Bot
ai crawler
Echobox
EchoboxBot
observe
ecoresearchCrawler
other
ecoresearchCrawler
observe
EcoVadisSustainabilityBot
ai assistant
EcoVadis
EcoVadisSustainabilityBot
observe
edisterbot
other
edisterbot
observe
EFF DNT policy checker
ai coding agent
Electronic Frontier
observe
ELB-HealthChecker
ai coding agent
Amazon
ELB-HealthChecker
observe