Updated daily

Crawler Catalog

1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

471

Allowed

1,284

Observed

198

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

All 1,953 Search Engine 118 AI Crawler 69 AI Assistant 290 AI Agent 11 AI Coding Agent 75 Browser Automation 2 AI Data Provider 13 AI Scraper 41 SEO Crawler 193 Social Media 73 Security 81 Monitoring 122 Archiver 21 Other 844

1,953 results

CherryPicker other

CherryPicker

observe

CherryPickerElite other

ChimeUnfurlAgent other

ChimeUnfurlAgent

observe

Chirp ai coding agent

Binary Canary

Chirp

observe

Chrome-Lighthouse seo crawler

Google

CINC

cis5550-crawler other

CISPA Webcrawler other

CitibotSiteCrawler ai crawler

Citibot

Wikimedia

ClarityBot seo crawler

seoClarity

Anthropic

Claude-Code ai coding agent

Anthropic

Claude-Code

observe

Claude-SearchBot ai crawler

Anthropic

Claude-SearchBot

observe

Claude-User ai assistant

Anthropic

Anthropic

Anthropic

ClearscopeBot seo crawler

Mushi Labs

Clearscopebot

challenge

Cledara SaaS Management Agent ai assistant

Cledara

CledaraBot

observe signed

Clickagy seo crawler

Clickagy

Clickagy Intelligence Bot v2

challenge

Clickagy Intelligence Bot ai assistant

Clickagy

observe

Clickagy Intelligence Bot v2 ai assistant

Clickagy

observe

ClickUpLinkUnfurler ai assistant

ClickUp

ClickUpLinkUnfurler

Cloudflare AutoRAG ai crawler

Cloudflare

Cloudflare-AutoRAG

observe

Cloudflare Browser Run ai assistant

Cloudflare

cloudflare-browser-rendering

observe signed

Cloudflare Crawler ai crawler

Cloudflare

Cloudflare Custom Hostname Verification

observe signed

Cloudflare CSUP other

Cloudflare

cloudflare-csup

observe

Cloudflare Custom Error Page Crawler seo crawler

Cloudflare

challenge

Cloudflare Custom Hostname Verification seo crawler

challenge

Cloudflare CustomerSLO Prober monitoring

Cloudflare

allow

Cloudflare Diagnostics monitoring

Cloudflare

CloudflareDiagnostics

allow

Cloudflare Digicert DCV seo crawler

Cloudflare

challenge

Cloudflare Healthchecks seo crawler

Cloudflare

Cloudflare-Healthchecks

challenge

Cloudflare Observatory Curly monitoring

Cloudflare

curly-

allow

Cloudflare Prefetch seo crawler

Cloudflare

CloudFlare-Prefetch

challenge

Cloudflare Purge seo crawler

Cloudflare

challenge

Cloudflare Radar URL Scanner security

Cloudflare

cloudflare-radar-url-scanner

allow

Cloudflare SpeedTest seo crawler

Cloudflare

Cloudflare SpeedTest

challenge

Cloudflare SSLDetector seo crawler

Cloudflare

Cloudflare-SSLDetector

challenge

Cloudflare Stream Webhook seo crawler

Cloudflare

Cloudflare Stream Webhook

challenge

Cloudflare Validator other

Cloudflare

Cloudflare-Validator

observe

Cloudflare Web Scanner search engine

Cloudflare

CloudflareWebScanner

allow

CloudFlare-AlwaysOnline other

CloudFlare-AlwaysOnline

observe

cloudflare-csup other

cloudflare-csup

observe

Cloudflare-Healthchecks seo crawler

Cloudflare-Healthchecks

challenge

Cloudflare-SSLDetector seo crawler

Cloudflare-SSLDetector

challenge

Cloudflare-Traffic-Manager seo crawler

Cloudflare

Cloudflare-Traffic-Manager

challenge

CloudflareDiagnostics other

CloudflareDiagnostics

observe

Cloudtrellis monitoring

Cloudtrellis

Cloudtrellis

allow

CloudVertexBot ai scraper

Google

Cludo

cludo.com seo crawler

Cludo

cludo.com

challenge

cludo.com bot search engine

Cludo

cludo.com

allow

ClueWeb-Crawler other

← Prev 6 / 33 Next →