Skip to main content
Updated daily

Crawler Catalog

1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

460

Allowed

1,224

Observed

192

Challenged

0

Blocked
All 1,876 Search Engine 114 AI Crawler 64 AI Assistant 268 AI Agent 10 AI Coding Agent 75 Browser Automation 3 AI Data Provider 13 AI Scraper 40 SEO Crawler 187 Social Media 73 Security 74 Monitoring 122 Archiver 21 Other 812

1,876 results

CitibotSiteCrawler ai crawler

Citibot

observe
Citoid ai assistant

Wikimedia

Citoid
observe
ClarityBot seo crawler

seoClarity

ClarityBot
challenge
Claude ai crawler

Anthropic

observe
Claude-Code ai coding agent

Anthropic

Claude-Code
observe
Claude-SearchBot ai crawler

Anthropic

Claude-SearchBot
observe
Claude-User ai assistant

Anthropic

Claude-User
observe
Claude-Web ai agent

Anthropic

Claude-Web
observe
ClaudeBot ai scraper

Anthropic

ClaudeBot
observe
ClearscopeBot seo crawler

Mushi Labs

Clearscopebot
challenge
Cledara SaaS Management Agent ai assistant

Cledara

observe signed
Clickagy seo crawler

Clickagy

Clickagy
challenge
ClickUpLinkUnfurler ai assistant

ClickUp

ClickUpLinkUnfurler
observe
Cliqzbot other
Cliqzbot
observe
Cloudflare AutoRAG ai crawler

Cloudflare

Cloudflare-AutoRAG
observe
Cloudflare Browser Run ai assistant

Cloudflare

Cloudflare
observe signed
Cloudflare Crawler ai crawler

Cloudflare

Cloudflare
observe signed
Cloudflare CSUP other

Cloudflare

cloudflare-csup
observe
Cloudflare Custom Error Page Crawler seo crawler

Cloudflare

challenge
Cloudflare Custom Hostname Verification seo crawler
challenge
Cloudflare CustomerSLO Prober monitoring

Cloudflare

allow
Cloudflare Diagnostics monitoring

Cloudflare

CloudflareDiagnostics
allow
Cloudflare Digicert DCV seo crawler

Cloudflare

challenge
Cloudflare Healthchecks seo crawler

Cloudflare

Cloudflare-Healthchecks
challenge
Cloudflare Observatory Curly monitoring

Cloudflare

allow
Cloudflare Prefetch seo crawler

Cloudflare

CloudFlare-Prefetch
challenge
Cloudflare Purge seo crawler

Cloudflare

challenge
Cloudflare Radar URL Scanner security

Cloudflare

Cloudflare
allow
Cloudflare SpeedTest seo crawler

Cloudflare

Cloudflare
challenge
Cloudflare SSLDetector seo crawler

Cloudflare

Cloudflare-SSLDetector
challenge
Cloudflare Stream Webhook seo crawler

Cloudflare

Cloudflare
challenge
Cloudflare Validator other

Cloudflare

Cloudflare-Validator
observe
Cloudflare Web Scanner search engine

Cloudflare

allow
CloudFlare-AlwaysOnline other
CloudFlare-AlwaysOnline
observe
Cloudflare-Traffic-Manager seo crawler

Cloudflare

Cloudflare-Traffic-Manager
challenge
Cloudtrellis monitoring

Cloudtrellis

Cloudtrellis
allow
CloudVertexBot ai scraper

Google

CloudVertexBot
observe
Cludo monitoring

Cludo

Cludo
allow
cludo.com search engine

Cludo

cludo.com
allow
ClueWeb-Crawler other
ClueWeb-Crawler
observe
CMS-Checker other
CMS-Checker
observe
CMU-Cylab ai scraper

Carnegie Mellon University

observe
Cốc Cốc search engine

Coccoc

coccocbot
allow
coccoc other
coccoc
observe
coccocbot-image search engine

Coc Coc

coccocbot-image
allow
coccocbot-web search engine

Coc Coc

coccocbot-web
allow
Cocolyzebot seo crawler

Cocolyze

Cocolyzebot
challenge
CodaBot other
CodaBot
observe
Code ai coding agent

GitHub

Code
observe
coexel other
coexel
observe
cognitiveSEO Crawler seo crawler

cognitive SEO Internet Marketing Tools.

cognitiveSEO
challenge
Cohere Training Data Crawler ai crawler

Cohere

observe
cohere-ai ai agent

Cohere

cohere-ai
observe
cohere-training-data-crawler ai scraper

Cohere

cohere-training-data-crawler
observe
CohereBot ai scraper

Cohere

CohereBot
observe
Coinbase Webhooks other

Coinbase

allow
Collapsify ai assistant

Cloudflare

Collapsify
observe
colly other
colly
observe
CommaFeed other
CommaFeed
observe
Companybook-Crawler other
Companybook-Crawler
observe
← Prev 6 / 32 Next →