Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,953 results
Crescent Internet ToolPak HTTP OLE Control v.1.0
other
observe
CriteoBot
seo crawler
Criteo
CriteoBot
challenge
Critical CSS Bot
other
Critical CSS
Critical
observe
Criticalcss.com
ai coding agent
Critical CSS
Criticalcss.com
observe
cron-job.org
other
cron-job.org
cron-job.org
allow
CrunchBot
other
CrunchBot
observe
CrystalSemanticsBot
other
CrystalSemanticsBot
observe
CUCH.org Archive Bot
archiver
C U Cyber History
CUCH.org
allow
Curebot
other
Curebot
observe
Current
ai assistant
Current
Current
observe
Cursor
ai coding agent
xAI
Cursor
observe
Cutbot
other
Cutbot
observe
Cxense
seo crawler
Cxense
cXensebot
challenge
Cybaa Bot
security
CybaaBot
allow
CybaaBot
other
Cybaa
CybaaBot
observe
CyberFindCrawler
other
CyberFindCrawler
observe
CyberOKInspect
security
CyberOK
CyberOKInspect
allow
CyberPatrol
other
CyberPatrol
observe
CyotekWebCopy
other
CyotekWebCopy
observe
cypex.ai/scanning
security
Cypex
cypex.ai/scanning
allow
d-bot-health-check
other
d-bot-health-check
observe
DareBoost
other
DareBoost
observe
Daric2
ai crawler
Cloudflare
daric2
observe
Daric3
ai crawler
Cloudflare
daric3
observe
Daric4
ai crawler
Cloudflare
daric4
observe
Dark Visitor
other
Dark
observe
Datadog
ai coding agent
Datadog
Datadog Agent
observe
Datadog Synthetic
ai coding agent
Datadog
observe
Datafeedwatch
other
Datafeedwatch
observe
DataForSEO
seo crawler
DataForSEO
DataForSEO
challenge
DataForSEO Bot
seo crawler
DataForSEO
RankMath Link Checker
challenge
DataForSeoBot
seo crawler
DataForSEO
DataForSeoBot
challenge
datagnionbot
other
datagnionbot
observe
Datanyze
other
Datanyze
observe
Dataprovider
search engine
Dataprovider.com
Dataprovider
allow
Dataprovider.com
search engine
Dataprovider.com
Dataprovider
allow
Datenbank Crawler
ai scraper
netEstate
Datenbank
observe
Daum
search engine
Daum
Daum
allow
Daumoa
other
Daumoa
observe
Dazzle BlueSky Bot
other
Dazzle
observe
dbot
other
dbot
observe
dcrawl
seo crawler
dcrawl
challenge
deadlinkchecker
monitoring
DLC Websites
deadlinkchecker
allow
Deck Agent
ai assistant
DeckAgent
observe
signed
Deepcrawl
monitoring
Lumar
deepcrawl
allow
deepcrawl.com
seo crawler
Lumar
deepcrawl.com
challenge
deepnoc
other
deepnoc
observe
DeepSeekBot
ai crawler
DeepSeek
DeepSeekBot
observe
Detectify
security
Detectify
Detectify
allow
DeuSu
other
DeuSu
observe
Devin
ai assistant
Devin AI
Devin
observe
Diffbot
ai data provider
Diffbot
Diffbot
observe
Digg Deeper
other
Digg
observe
DigiCert DCV
seo crawler
DigiCert
Digicert
challenge
Digimind
other
Digimind
observe
Digincore bot
other
Digincore
observe
Direqt Anomura
ai crawler
Direqt
Anomura
observe
discobot
other
discobot
observe
discordBot
social media
Discord, Inc.
Discordbot
allow
Disqus
other
Disqus
observe