Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,426 results
Crescent Internet ToolPak HTTP OLE Control v.1.0
other
observe
Critical CSS Bot
other
Critical CSS
Critical
observe
Criticalcss.com
ai coding agent
Critical CSS
Criticalcss.com
observe
CrunchBot
other
CrunchBot
observe
CrystalSemanticsBot
other
CrystalSemanticsBot
observe
CT-WP-Probe
other
CT-WP-Probe
observe
Curebot
other
Curebot
observe
Current
ai assistant
Current
Current
observe
Cursor
ai coding agent
xAI
Cursor
observe
Cutbot
other
Cutbot
observe
CybaaBot
other
Cybaa
CybaaBot
observe
CyberFindCrawler
other
CyberFindCrawler
observe
CyberPatrol
other
CyberPatrol
observe
CyotekWebCopy
other
CyotekWebCopy
observe
CypressGrowthDiligence
ai assistant
CypressGrowthDiligence
observe
d-bot-health-check
other
d-bot-health-check
observe
DareBoost
other
DareBoost
observe
Daric2
ai crawler
Cloudflare
daric2
observe
Daric3
ai crawler
Cloudflare
daric3
observe
Daric4
ai crawler
Cloudflare
daric4
observe
Dark Visitor
other
Dark
observe
Datadog
ai coding agent
Datadog
Datadog Agent
observe
Datadog Synthetic
ai coding agent
Datadog
observe
Datafeedwatch
other
Datafeedwatch
observe
datagnionbot
other
datagnionbot
observe
Datanyze
other
Datanyze
observe
Datenbank Crawler
ai scraper
netEstate
Datenbank
observe
Daumoa
other
Daumoa
observe
Dazzle BlueSky Bot
other
Dazzle
observe
DBD Bot
other
observe
dbot
other
dbot
observe
Deck Agent
ai assistant
DeckAgent
observe
signed
deepnoc
other
deepnoc
observe
DeepSeekBot
ai crawler
DeepSeek
DeepSeekBot
observe
DesearchBot
ai data provider
Desearch
DesearchBot
observe
DeuSu
other
DeuSu
observe
Devin
ai assistant
Devin AI
Devin
observe
Diffbot
ai data provider
Diffbot
Diffbot
observe
Digg Deeper
other
Digg
observe
Digimind
other
Digimind
observe
Digincore bot
other
Digincore
observe
Direqt Anomura
ai crawler
Direqt
Anomura
observe
discobot
other
discobot
observe
Disqus
other
Disqus
observe
DittoSpyder
other
DittoSpyder
observe
DLC
ai coding agent
Dr. Link Check
DLC
observe
dlvr.it
ai assistant
dlvr.it
dlvr.it
observe
Dmbot
other
Dmbot
observe
DMBrowser
ai coding agent
Dotcom-Monitor
DMBrowser
observe
DnBCrawler-Analytics
other
DnBCrawler-Analytics
observe
DNSCrawler
other
DNSCrawler
observe
DNSScanner
other
DNSScanner
observe
DnyzBot
other
DnyzBot
observe
Do Not Track Verifier
ai coding agent
Electronic Frontier Foundation
observe
Dola-SearchBot
ai crawler
ByteDance
Dola-SearchBot
observe
Domain Re-Animator Bot
other
Domain
observe
DomainCrawler
other
DomainCrawler
observe
DomainDrift
ai assistant
DRM3
DomainDrift
observe
DomainEvidenceBot
other
DomainEvidenceBot
observe
DomainMapper
other
DomainMapper
observe