Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,426 results
keys-so-bot
other
keys-so-bot
observe
Keyword Density
other
Keyword
observe
KimeBot
ai assistant
KIME
KimeBot
observe
Kimi-Agent
ai agent
Moonshot AI
Kimi-Agent
observe
Kimi-SearchBot
ai crawler
Moonshot AI
Kimi-SearchBot
observe
Kimi-User
ai assistant
Moonshot AI
Kimi-User
observe
KimiBot
ai crawler
Moonshot AI
KimiBot
observe
KimiCrawler
ai scraper
Moonshot AI
KimiCrawler
observe
KlaviyoAIBot
ai crawler
Klaviyo
KlaviyoAIBot
observe
Knowings
other
Knowings
observe
Known Agent
ai assistant
Known Agents
Known
observe
signed
Known Agents Browser
other
Known Agents
observe
KOCMOHABT
other
KOCMOHABT
observe
KomodiaBot
other
KomodiaBot
observe
Konqueror
other
Konqueror
observe
Konturbot
ai assistant
Kontur
Konturbot
observe
KosmioBot
other
KosmioBot
observe
KStandBot
ai assistant
URL Classification
KStandBot
observe
KunatoCrawler
ai agent
Zzazz
KunatoCrawler
observe
l9scan
other
l9scan
observe
laion-huggingface-processor
ai crawler
LAION / Hugging Face
laion-huggingface-processor
observe
LamarkBot
other
LamarkBot
observe
Landau-Media-Spider
other
Landau-Media-Spider
observe
larbin
other
larbin
observe
Laserlikebot
other
Laserlikebot
observe
lb-spider
other
lb-spider
observe
LCC
ai scraper
University of Leipzig
LCC
observe
leadbox
other
leadbox
observe
LeadFinderBot
ai assistant
LeadFinderBot
observe
LeadForge
ai assistant
NewSiteLead
LeadForge
observe
Let's Encrypt
ai coding agent
Let's Encrypt
Let&
observe
LexiBot
other
LexiBot
observe
Library Of Congress Web Archiving
ai scraper
United States Library of Congress
Library
observe
libWeb
other
libWeb
observe
Light Crawler
other
Light
observe
LighthouseBot
ai assistant
Lighthouse
LighthouseBot
observe
LightspeedSystemsCrawler
ai assistant
Lightspeed Systems
LightspeedSystemsCrawler
observe
LINER Bot
ai crawler
Liner Bot
LinerBot
observe
LinerBot
ai assistant
Liner
LinerBot
observe
Linguee Bot
other
Linguee
observe
Link CLI
ai assistant
Link
link-cli
observe
signed
linkapediabot
other
linkapediabot
observe
LinkArchiver
other
LinkArchiver
observe
LinkCheck by Siteimprove.com
other
observe
linkdex
other
linkdex
observe
LinkextractorPro
other
LinkextractorPro
observe
linkfluence
ai assistant
Meltwater
linkfluence
observe
Linkhub-crawler
other
Linkhub-crawler
observe
LinkisBot
other
LinkisBot
observe
linko
other
linko
observe
LinkpadBot
other
LinkpadBot
observe
LinkPreview
ai assistant
Interactive32
LinkPreview
observe
LinkRing
other
LinkRing
observe
LinkScan
other
LinkScan
observe
LinkupBot
ai crawler
Linkup
LinkupBot
observe
LinkWalker
ai assistant
Fortra
LinkWalker
observe
lipperhey
other
lipperhey
observe
ListSignalBot
ai assistant
ListSignal
ListSignalBot
observe
LivelapBot
other
LivelapBot
observe
lkxscan
other
lkxscan
observe