Crawler Catalog
1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
64 results · AI Crawler
Cloudflare
Amazon
amazon-kendra
Amazon
Amazonbot
Amazon
Amzn-SearchBot
Anchor
Anchor
Direqt
Anomura
Apple
Applebot
Atlassian
Atlassian
atlassian-bot
Awario
Awario
Microsoft
AzureAI-SearchBot
Big Sur AI
bigsur.ai
Borderxlab (Nubestore AI)
Brandwatch
Brave Software, Inc.
Bravebot
Citibot
Anthropic
Anthropic
Claude-SearchBot
Cloudflare
Cloudflare-AutoRAG
Cloudflare
Cloudflare
Cohere
Research Organization of Information and Systems
Cotoyogi
Open-source
Crawl4AI
Cloudflare
Cloudflare
Cloudflare
DeepSeek
DeepSeekBot
Direqt
Echobox
EchoboxBot
Factset
factset_spyderbot
Mendable / Firecrawl
FirecrawlAgent
FishBot
Google-CloudVertexBot
Google-NotebookLM
GoogleOther
OpenAI
GPTBot
NICT
Moonshot AI
Kimi-SearchBot
Moonshot AI
KimiBot
Klaviyo
KlaviyoAIBot
LAION / Hugging Face
laion-huggingface-processor
Liner Bot
Linkup
LinkupBot
Make.com
Meta
meta-externalagent
Mistral
MistralAI-Index
netEstate
Novellum
OpenAI
OAI-SearchBot
ADP
Perplexity
PerplexityBot
Huawei
PetalBot
Qualified.com, Inc.
QualifiedBot
Semrush
SemrushBot-OCOB
Semrush
SemrushBot-SWA
Parallel
ShapBot
Spawning
Tavily
TavilyBot
WEBSPARK
WARDBot
xAI
xAI-SearchBot