Skip to main content
Updated daily

Crawler Catalog

1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

460

Allowed

1,224

Observed

192

Challenged

0

Blocked
Clear
All 1,876 Search Engine 114 AI Crawler 64 AI Assistant 268 AI Agent 10 AI Coding Agent 75 Browser Automation 3 AI Data Provider 13 AI Scraper 40 SEO Crawler 187 Social Media 73 Security 74 Monitoring 122 Archiver 21 Other 812

40 results · AI Scraper

AI2Bot ai scraper

Ai2

AI2Bot
observe
Ai2Bot-Dolma ai scraper

Ai2

Ai2Bot-Dolma
observe
Applebot-Extended ai scraper

Apple

Applebot-Extended
observe
bedrockbot ai scraper

Amazon

bedrockbot
observe
Bibliotheque Nacional de France Crawler ai scraper

Bibliothèque nationale de France

observe
Bytespider ai scraper

ByteDance

Bytespider
observe
CCBot ai scraper

Common Crawl

CCBot
observe
ChatGLM-Spider ai scraper

Zhipu AI

ChatGLM-Spider
observe
ClaudeBot ai scraper

Anthropic

ClaudeBot
observe
CloudVertexBot ai scraper

Google

CloudVertexBot
observe
CMU-Cylab ai scraper

Carnegie Mellon University

observe
cohere-training-data-crawler ai scraper

Cohere

cohere-training-data-crawler
observe
CohereBot ai scraper

Cohere

CohereBot
observe
Datenbank ai scraper

netEstate

Datenbank
observe
Doubaobot ai scraper

ByteDance

Doubaobot
observe
ERNIEBot ai scraper

Baidu

ERNIEBot
observe
FacebookBot ai scraper

Meta

FacebookBot
observe
Google-Extended ai scraper

Google

Google-Extended
observe
ICC-Crawler ai scraper

NICT

ICC-Crawler
observe
imageSpider ai scraper

ByteDance

imageSpider
observe
Kangaroo ai scraper

Kangaroo LLM

Kangaroo
observe
kb.dk_bot ai scraper

Netarkivet

kb.dk_bot
observe
KimiCrawler ai scraper

Moonshot AI

KimiCrawler
observe
LCC ai scraper

University of Leipzig

LCC
observe
Library Of Congress Web Archiving ai scraper

United States Library of Congress

Library
observe
micro-crawl ai scraper

Reflection AI

micro-crawl
observe
MistralBot ai scraper

Mistral

MistralBot
observe
MoonshotBot ai scraper

Moonshot AI

MoonshotBot
observe
MoonSpider ai scraper

Moonshot AI

MoonSpider
observe
netEstate ai scraper

netEstate

netEstate
observe
omgili ai scraper

Webz.io

omgili
observe
PanguBot ai scraper

Huawei

PanguBot
observe
QwenBot ai scraper

Alibaba

QwenBot
observe
SalamandraVLM ai scraper

Aithlas

SalamandraVLM
observe
SBIntuitionsBot ai scraper

SB Intuitions

SBIntuitionsBot
observe
Spider ai scraper

Spider

Spider
observe
Timpibot ai scraper

Timpi

Timpibot
observe
TurnitinBot ai scraper

Turnitin

TurnitinBot
observe
VelenPublicWebCrawler ai scraper

Velen

VelenPublicWebCrawler
observe
webzio-extended ai scraper

Webz.io

webzio-extended
observe