Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
2,114 results
Cosmos
Research Organization of Information and Systems
Cotoyogi
Coveo
Coveobot
OurFamilyWizard
Cozi-iCalendar-FeedReader
CragSoftware
CragCrawler
Open-source
Crawl4AI
CrawlBot
crawler_eb_germany
Freespoke
crawler.freespoke.com
crawler4j
Crawlson
Crawlson
Crawlspace
Crawlspace
CrawlyProjectCrawler
CrazyEgg
Crazy
Crescent
Criteo
CriteoBot
Critical CSS
Critical
Critical CSS
Criticalcss.com
cron-job.org
cron-job.org
CrunchBot
CrystalSemanticsBot
CT-WP-Probe
C U Cyber History
CUCH.org
Curebot
Current
Current
xAI
Cursor
Cutbot
Cxense
cXensebot
CybaaBot
Cybaa
CybaaAgent
Cybaa
CybaaBot
CyberConvoy
CyberConvoyScout
CyberFindCrawler
CyberOK
CyberOKInspect
CyberPatrol
CyotekWebCopy
Cypex
cypex.ai/scanning
CypressGrowthDiligence
d-bot-health-check
DareBoost
Cloudflare
daric2
Cloudflare
daric3
Cloudflare
daric4
Dark
Datadog
Datadog Agent
Datadog
Datafeedwatch
DataForSEO
RankMath Link Checker
DataForSEO
DataForSeoBot
datagnionbot
Datanyze
Dataprovider.com
Dataprovider
Dataprovider.com
Dataprovider
netEstate
Datenbank
Daum
Daum
Daumoa
Dazzle
dbot