Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,953 results
DittoSpyder
Dr. Link Check
DLC
dlvr.it
dlvr.it
dlvr.it
Dotcom-Monitor
DMBrowser
DnBCrawler-Analytics
DNSCrawler
DNSScanner
DnyzBot
Electronic Frontier Foundation
DocSpot
DocSpotBot
Dotcom-Monitor
DMBrowser
Domain
DomainCrawler
DomainMapper
Domains
Game Lounge
DomainStatsBot
DominicBot
ByteDance
Doubaobot
Download
Dr. Link Check
Dragon Metrics
Dragonbot
Publisher Discovery
Dragonfly
Drata
Dratabot
Drata
Dratabot
DreamHost
DreamHost
Drift Security
drift-agent
drupact
Drupalbot
Copyright Clearance Center
ds9
DubBot
Dubbotbot
DuckDuckGo
DuckAssistBot
DuckDuckGo
DuckDuckBot
DuckDuckGo
DuckDuckGo-Favicons-Bot
DoubleVerify
DVbot
e.ventures
EasyBib
easybill.de
easybill-ImportManager
easybill
easybill-ImportManager
EasyCron
EasyDNS
easyDNS
codire GmbH
EasyScan
ec2linkfinder
Echobox
EchoboxBot
ecoresearchCrawler
EcoVadis
EcoVadisSustainabilityBot
edisterbot
Electronic Frontier
Amazon
ELB-HealthChecker
electricmonk
Elisabot
ellisphere
elmah.io
elmahio-uptimebot
EmailCollector
EmailSiphon
EmailWolf
Embedly
Embedly
emetriqContextualBot