Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
2,114 results
DuckAssistbot
ai assistant
DuckDuckGo
DuckAssistBot
observe
DuckDuckBot
search engine
DuckDuckGo
DuckDuckBot
allow
DuckDuckGo-Favicons-Bot
search engine
DuckDuckGo
DuckDuckGo-Favicons-Bot
allow
DVbot
ai assistant
DoubleVerify
DVbot
observe
dwelltime-bot
ai assistant
Dwell
dwelltime-bot
observe
e.ventures Investment Crawler
other
e.ventures
observe
EasyBib AutoCite
other
EasyBib
observe
EasyBill Import Manager
other
easybill.de
easybill-ImportManager
observe
EasyCron
other
EasyCron
allow
Easydns
monitoring
EasyDNS
easyDNS
allow
easyDNS Monitoring
other
observe
EasyScan
ai assistant
codire GmbH
EasyScan
observe
signed
ec2linkfinder
other
ec2linkfinder
observe
Echobot Bot
ai crawler
Echobox
EchoboxBot
observe
ecoresearchCrawler
other
ecoresearchCrawler
observe
EcoVadisSustainabilityBot
ai assistant
EcoVadis
EcoVadisSustainabilityBot
observe
edisterbot
other
edisterbot
observe
EFF DNT policy checker
ai coding agent
Electronic Frontier
observe
ELB-HealthChecker
ai coding agent
Amazon
ELB-HealthChecker
observe
electricmonk
other
electricmonk
observe
ElevenlabsBot
ai assistant
ElevenLabs
ElevenlabsBot
observe
ElevenlabsCrawler
ai scraper
ElevenLabs
ElevenlabsCrawler
observe
Elisabot
other
Elisabot
observe
ellisphere
other
ellisphere
observe
elmah.io Uptime Monitoring
monitoring
elmah.io
elmahio-uptimebot
allow
EmailCollector
other
EmailCollector
observe
EmailSiphon
other
EmailSiphon
observe
EmailWolf
other
EmailWolf
observe
eManagerBot
other
eManagerBot
observe
Embedly
social media
Embedly
Embedly
allow
emetriqContextualBot
other
emetriqContextualBot
observe
eMoney Advisor
seo crawler
eMoney Advisor
eMoneyBot
challenge
epicbot
other
epicbot
observe
Epivoz Crawler
seo crawler
Techmeme
challenge
EpivozCrawler
ai assistant
Techmeme
EpivozCrawler
observe
eRepublik.tools
monitoring
Sebastian Foth - Software Solutions
eRepublik.tools
allow
eright
other
eright
observe
EriounBot
other
Erioun
EriounBot
observe
ErisBot
other
ErisBot
observe
ERNIEBot
ai scraper
Baidu
ERNIEBot
observe
EroCrawler
other
EroCrawler
observe
EtaoSpider
other
EtaoSpider
observe
EtherdeckBot
search engine
Etherdeck
EtherdeckBot
allow
europarchive.org
other
europarchive.org
observe
ev-crawler
ai assistant
Headline
ev-crawler
observe
evc-batch
other
evc-batch
observe
EvernoteRichLinkBot
ai assistant
Evernote
EvernoteRichLinkBot
observe
everyfeed-spider
other
everyfeed-spider
observe
EveryoneSocialBot
other
EveryoneSocialBot
observe
EvoUptimeBot
monitoring
Evo Agency Ltd.
EvoUptimeBot
allow
ExaBot
ai data provider
Exa
ExaBot
observe
ExaSearchBot
ai crawler
Exa
ExaSearchBot
observe
ExaUser
ai assistant
Exa
ExaUser
observe
Exodus
other
Exodus
ExodusMovement
allow
Expanse
security
Palo Alto Networks
Expanse
allow
Experibot
other
Experibot
observe
expo-research-bot
other
expo-research-bot
observe
ExposureWatch-Benchmark
other
ExposureWatch-Benchmark
observe
ExteContextCrawl
other
ExteContextCrawl
observe
ExtLinksBot
other
ExtLinksBot
observe