Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,426 results
Domains Project
other
Domains
observe
DominicBot
other
DominicBot
observe
Dot
ai agent
OpenAI
observe
DotSee-HealthCheck
ai coding agent
DotSee
DotSee-HealthCheck
observe
Doubaobot
ai scraper
ByteDance
Doubaobot
observe
Dow Jones Searchbot
other
observe
Download Ninja
other
Download
observe
DowntimeDetector
ai coding agent
UpDownSideways
DowntimeDetector
observe
Dragonfly
ai assistant
Publisher Discovery
Dragonfly
observe
DreamHost Data Team
ai coding agent
DreamHost
DreamHost
observe
Drift Agent
ai assistant
Drift Security
drift-agent
observe
signed
drupact
other
drupact
observe
Drupalbot
other
Drupalbot
observe
ds9
ai assistant
Copyright Clearance Center
ds9
observe
Dubbotbot
ai coding agent
DubBot
Dubbotbot
observe
DuckAssistbot
ai assistant
DuckDuckGo
DuckAssistBot
observe
DVbot
ai assistant
DoubleVerify
DVbot
observe
dwelltime-bot
ai assistant
Dwell
dwelltime-bot
observe
e.ventures Investment Crawler
other
e.ventures
observe
EasyBib AutoCite
other
EasyBib
observe
EasyBill Import Manager
other
easybill.de
easybill-ImportManager
observe
easyDNS Monitoring
other
observe
EasyScan
ai assistant
codire GmbH
EasyScan
observe
signed
ec2linkfinder
other
ec2linkfinder
observe
Echobot Bot
ai crawler
Echobox
EchoboxBot
observe
ecoresearchCrawler
other
ecoresearchCrawler
observe
EcoVadisSustainabilityBot
ai assistant
EcoVadis
EcoVadisSustainabilityBot
observe
edisterbot
other
edisterbot
observe
EFF DNT policy checker
ai coding agent
Electronic Frontier
observe
ELB-HealthChecker
ai coding agent
Amazon
ELB-HealthChecker
observe
electricmonk
other
electricmonk
observe
ElevenlabsBot
ai assistant
ElevenLabs
ElevenlabsBot
observe
ElevenlabsCrawler
ai scraper
ElevenLabs
ElevenlabsCrawler
observe
Elisabot
other
Elisabot
observe
ellisphere
other
ellisphere
observe
EmailCollector
other
EmailCollector
observe
EmailSiphon
other
EmailSiphon
observe
EmailWolf
other
EmailWolf
observe
eManagerBot
other
eManagerBot
observe
emetriqContextualBot
other
emetriqContextualBot
observe
epicbot
other
epicbot
observe
EpivozCrawler
ai assistant
Techmeme
EpivozCrawler
observe
eright
other
eright
observe
EriounBot
other
Erioun
EriounBot
observe
ErisBot
other
ErisBot
observe
ERNIEBot
ai scraper
Baidu
ERNIEBot
observe
EroCrawler
other
EroCrawler
observe
EtaoSpider
other
EtaoSpider
observe
europarchive.org
other
europarchive.org
observe
ev-crawler
ai assistant
Headline
ev-crawler
observe
evc-batch
other
evc-batch
observe
EvernoteRichLinkBot
ai assistant
Evernote
EvernoteRichLinkBot
observe
everyfeed-spider
other
everyfeed-spider
observe
EveryoneSocialBot
other
EveryoneSocialBot
observe
ExaBot
ai data provider
Exa
ExaBot
observe
ExaSearchBot
ai crawler
Exa
ExaSearchBot
observe
ExaUser
ai assistant
Exa
ExaUser
observe
Experibot
other
Experibot
observe
expo-research-bot
other
expo-research-bot
observe
ExposureWatch-Benchmark
other
ExposureWatch-Benchmark
observe