Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
69 results · AI Crawler
Cloudflare
Cloudflare-AI-Search
Lyrenth
AIWebIndex-Agent
Amazon
amazon-kendra
Amazon
Amazonbot
Amazon
Amzn-SearchBot
Anchor
Anchor
Direqt
Anomura
Apple
Applebot
Atlassian
atlassian-bot
Atlassian
atlassian-bot
Awario
AwarioSmartBot
Microsoft
AzureAI-SearchBot
Big Sur AI
bigsur.ai
Borderxlab (Nubestore AI)
BorderxBot
Brandwatch
magpie-crawler
Brave Software, Inc.
Bravebot
Citibot
CitibotSiteCrawler
Anthropic
ClaudeBot
Anthropic
Claude-SearchBot
Cloudflare
Cloudflare-AutoRAG
Cloudflare
Cloudflare Custom Hostname Verification
Cohere
cohere-training-data-crawler
Research Organization of Information and Systems
Cotoyogi
Open-source
Crawl4AI
Cloudflare
daric2
Cloudflare
daric3
Cloudflare
daric4
DeepSeek
DeepSeekBot
Direqt
Anomura
Echobox
EchoboxBot
Exa
ExaSearchBot
Factset
factset_spyderbot
Mendable / Firecrawl
FirecrawlAgent
FishBot
FishBot
Google-CloudVertexBot
Google-NotebookLM
GoogleOther
OpenAI
GPTBot
NICT
Moonshot AI
Kimi-SearchBot
Moonshot AI
KimiBot
Klaviyo
KlaviyoAIBot
LAION / Hugging Face
laion-huggingface-processor
Liner Bot
LinerBot
Linkup
LinkupBot
Make.com
make.com
Meta
meta-externalagent
Meta
meta-webindexer
Mistral
MistralAI-Index
netEstate
netEstate Imprint Crawler
Novellum
Novellum
OpenAI
OAI-SearchBot
ADP
Perplexity
PerplexityBot
Huawei
PetalBot
Qualified.com, Inc.
QualifiedBot
Semrush
SemrushBot-OCOB
Semrush
SemrushBot-SWA
Parallel
ShapBot
Spawning
Spawning-AI