Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
2,114 results
kaikki.org-digital-archive
Kakao
kakaotalk-scrap
KamenX
Kangaroo LLM
Kangaroo
Kargo
KargoBot
Kargo
KargoBot-Artemis
Kargo
KargoBot-Artemis-Mobile
KauaiBot
Netarkivet
kb.dk_bot
kbcrawl
Keenable
Keenable-User
Keenable
KeenableBot
Kemvibot
Kenjin
Kernel
kernel
Keydrop.io
keys-so-bot
Keyword
KIME
KimeBot
Moonshot AI
Kimi-Agent
Moonshot AI
Kimi-SearchBot
Moonshot AI
Kimi-User
Moonshot AI
KimiBot
Moonshot AI
KimiCrawler
Kinsta
kinsta-bot
Klaviyo
Klaviyo/1.0
Klaviyo
KlaviyoAIBot
Knowings
Known Agents
Known
Known Agents
KOCMOHABT
KomodiaBot
Konqueror
Kontur
Konturbot
KosmioBot
Krawler
KrawlerBot
URL Classification
KStandBot
Zzazz
KunatoCrawler
l9scan
LAION / Hugging Face
laion-huggingface-processor
LamarkBot
Landau-Media-Spider
larbin
Laserlikebot
lb-spider
University of Leipzig
LCC
leadbox
LeadFinderBot
NewSiteLead
LeadForge
Legal Monster ApS
LegalMonster
DoubleVerify
Leikibot
Let's Encrypt
Let's Encrypt
Let&
Let's Encrypt
Let&
Level 9
Level9SearchBot
LexiBot
United States Library of Congress
Library
LibreCrawl
libWeb
Light