Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
41 results · AI Scraper
Ai2
AI2Bot
Ai2
Ai2Bot-Dolma
Apple
Applebot-Extended
Amazon
bedrockbot
Bibliothèque nationale de France
bnf.fr_bot
ByteDance
Bytespider
Common Crawl
CCBot
Zhipu AI
ChatGLM-Spider
Anthropic
ClaudeBot
CloudVertexBot
Carnegie Mellon University
CMU-Cylab
Cohere
cohere-training-data-crawler
Cohere
CohereBot
netEstate
Datenbank
ByteDance
Doubaobot
Baidu
ERNIEBot
Meta
FacebookBot
Google-Extended
NICT
ICC-Crawler
ByteDance
imageSpider
Kangaroo LLM
Kangaroo
Netarkivet
kb.dk_bot
Moonshot AI
KimiCrawler
LAION
laion-huggingface-processor
University of Leipzig
LCC
United States Library of Congress
Library
Reflection AI
micro-crawl
Mistral
MistralBot
Moonshot AI
MoonshotBot
Moonshot AI
MoonSpider
Webz.io
omgili
Huawei
PanguBot
Alibaba
QwenBot
Reflection
Reflectionbot
Aithlas
SalamandraVLM
SB Intuitions
SBIntuitionsBot
Spider
Spider
Timpi
Timpibot
Turnitin
TurnitinBot
Velen
VelenPublicWebCrawler
Webz.io
webzio-extended