Updated daily
Crawler Catalog
1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
1,876 results
Daric2
ai crawler
Cloudflare
observe
Daric3
ai crawler
Cloudflare
observe
Daric4
ai crawler
Cloudflare
observe
Dark
other
Dark
observe
Datadog
ai coding agent
Datadog
Datadog
observe
Datafeedwatch
other
Datafeedwatch
observe
DataForSEO Bot
seo crawler
DataForSEO
DataForSEO
challenge
DataForSeoBot
seo crawler
DataForSEO
DataForSeoBot
challenge
datagnionbot
other
datagnionbot
observe
Datanyze
other
Datanyze
observe
Dataprovider
search engine
Dataprovider.com
Dataprovider
allow
Dataprovider.com
search engine
Dataprovider.com
Dataprovider.com
allow
Datenbank
ai scraper
netEstate
Datenbank
observe
Daum
search engine
Daum
Daum
allow
Daumoa
other
Daumoa
observe
Dazzle
other
Dazzle
observe
dbot
other
dbot
observe
dcrawl
seo crawler
dcrawl
challenge
deadlinkchecker
monitoring
DLC Websites
deadlinkchecker
allow
Deck Agent
ai assistant
observe
signed
Deepcrawl
monitoring
Lumar
deepcrawl
allow
deepcrawl.com
seo crawler
Lumar
deepcrawl.com
challenge
deepnoc
other
deepnoc
observe
DeepSeekBot
ai crawler
DeepSeek
DeepSeekBot
observe
Detectify
security
Detectify
Detectify
allow
DeuSu
other
DeuSu
observe
Devin
ai assistant
Devin AI
Devin
observe
Diffbot
ai data provider
Diffbot
Diffbot
observe
Digg
other
Digg
observe
DigiCert
security
DigiCert
DigiCert
allow
DigiCert DCV
seo crawler
DigiCert
Digicert
challenge
Digimind
other
Digimind
observe
Digincore
other
Digincore
observe
Direqt Anomura
ai crawler
Direqt
observe
discobot
other
discobot
observe
discordBot
social media
Discord, Inc.
Discordbot
allow
Disqus
other
Disqus
observe
DittoSpyder
other
DittoSpyder
observe
DLC
ai coding agent
Dr. Link Check
DLC
observe
dlvr.it
ai assistant
dlvr.it
dlvr.it
observe
dlvr.it Feed Fetcher
seo crawler
dlvr.it
challenge
DMBrowser
ai coding agent
Dotcom-Monitor
DMBrowser
observe
DnBCrawler-Analytics
other
DnBCrawler-Analytics
observe
DNSCrawler
other
DNSCrawler
observe
DNSScanner
other
DNSScanner
observe
DnyzBot
other
DnyzBot
observe
Do
ai coding agent
Electronic Frontier Foundation
Do
observe
DocSpotBot
search engine
DocSpot
DocSpotBot
allow
Doctom
ai coding agent
Doctom
observe
Doctom Monitor
monitoring
Dotcom-Monitor
allow
Domain
other
Domain
observe
DomainCrawler
other
DomainCrawler
observe
DomainMapper
other
DomainMapper
observe
Domains
other
Domains
observe
DomainStatsBot
seo crawler
Game Lounge
DomainStatsBot
challenge
DominicBot
other
DominicBot
observe
Doubaobot
ai scraper
ByteDance
Doubaobot
observe
Dow
other
Dow
observe
Download
other
Download
observe
Dr. Link Check
monitoring
Dr. Link Check
allow