Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,953 results
keys-so-bot
Keyword
Moonshot AI
Kimi-SearchBot
Moonshot AI
Kimi-User
Moonshot AI
KimiBot
Moonshot AI
KimiCrawler
Kinsta
kinsta-bot
Kinsta
kinsta
Klaviyo
Klaviyo/1.0
Klaviyo
KlaviyoAIBot
Knowings
Known Agents
Known
Known Agents
KOCMOHABT
KomodiaBot
Konqueror
KosmioBot
Krawler
KrawlerBot
URL Classification
KStandBot
Zzazz
KunatoCrawler
l9scan
LAION / Hugging Face
laion-huggingface-processor
LAION
laion-huggingface-processor
LamarkBot
Landau-Media-Spider
larbin
Laserlikebot
lb-spider
University of Leipzig
LCC
leadbox
Legal Monster ApS
LegalMonster
DoubleVerify
Leikibot
Let's Encrypt
Let's Encrypt
Level 9
Level9SearchBot
LexiBot
United States Library of Congress
Library
LibreCrawl
libWeb
Light
Lighthouse
LighthouseBot
Lightspeed Systems
LightspeedSystemsCrawler
LINE
line-poker
Liner Bot
LinerBot
Liner
LinerBot
LINE Corporation
Linespider
Linguee
Link
link-cli
LinkTiger, Inc.
LinkTiger
linkapediabot
LinkArchiver
Siteimprove
LinkCheck
Siteimprove
LinkChecker
linkchecker.pro
LinkChecker
linkchecker.pro
LinkChecker.pro
LinkCheckerBot
linkdex
Linkdex
linkdexbot
LinkedInBot