Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
45 results · AI Scraper
Ai2
AI2Bot
Ai2
Ai2Bot-Dolma
Apple
Applebot-Extended
BeansLLM-CorpusBot
Amazon
bedrockbot
Bibliothèque nationale de France
bnf.fr_bot
ByteDance
Bytespider
Common Crawl
CCBot
Zhipu AI
ChatGLM-Spider
Anthropic
ClaudeBot
CloudVertexBot
Carnegie Mellon University
CMU-Cylab
Cohere
cohere-training-data-crawler
Cohere
CohereBot
netEstate
Datenbank
ByteDance
Doubaobot
ElevenLabs
ElevenlabsCrawler
Baidu
ERNIEBot
Meta
FacebookBot
Google-Extended
Tencent
Hunyuan
NICT
ICC-Crawler
ByteDance
imageSpider
Kangaroo LLM
Kangaroo
Netarkivet
kb.dk_bot
Moonshot AI
KimiCrawler
University of Leipzig
LCC
United States Library of Congress
Library
Reflection AI
micro-crawl
Mistral
MistralBot
Moonshot AI
MoonshotBot
Moonshot AI
MoonSpider
Webz.io
omgili
Huawei
PanguBot
Alibaba
QwenBot
Recraft
recraft-image-dataset
Reflection
Reflectionbot
Aithlas
SalamandraVLM
SB Intuitions
SBIntuitionsBot
Spider
Spider
Timpi
Timpibot
Turnitin
TurnitinBot
Velen
VelenPublicWebCrawler
Webz.io
webzio-extended
01.AI
YiBot