Updated daily
Crawler Catalog
1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
1,876 results
JobHuntBot
other
JobHuntBot
observe
Jobiglo-Bot
search engine
Jobiglo
Jobiglo-Bot
allow
jobswithgptcom-bot
search engine
JobsWithGPT
jobswithgptcom-bot
allow
Jooblebot
search engine
Jooble
Jooblebot
allow
jpg-newsbot
other
jpg-newsbot
observe
Jugendschutzprogramm-Crawler
ai assistant
JusProg
Jugendschutzprogramm-Crawler
observe
Jumio
security
Coinbase
Jumio
allow
jyxobot
other
jyxobot
observe
k2spider
other
k2spider
observe
K7MLWCBot
security
K7 Computing
K7MLWCBot
allow
Kagi Bot
search engine
Kagi
allow
kagi-fetcher
ai assistant
Kagi
kagi-fetcher
observe
Kagibot
search engine
Kagi
Kagibot
allow
kaikki.org-digital-archive
other
kaikki.org-digital-archive
observe
KakaoTalk Scrap
social media
Kakao
allow
Kangaroo
ai scraper
Kangaroo LLM
Kangaroo
observe
KargoBot
ai assistant
Kargo
KargoBot
observe
KargoBot-Artemis
monitoring
Kargo
KargoBot-Artemis
allow
KargoBot-Artemis-Mobile
ai assistant
Kargo
KargoBot-Artemis-Mobile
observe
KauaiBot
other
KauaiBot
observe
kb.dk_bot
ai scraper
Netarkivet
kb.dk_bot
observe
kbcrawl
other
kbcrawl
observe
Kemvibot
other
Kemvibot
observe
Kenjin
other
Kenjin
observe
Kernel
ai assistant
Kernel
observe
signed
Keydrop.io
other
Keydrop.io
observe
keys-so-bot
other
keys-so-bot
observe
Keyword
other
Keyword
observe
Kimi-SearchBot
ai crawler
Moonshot AI
Kimi-SearchBot
observe
Kimi-User
ai assistant
Moonshot AI
Kimi-User
observe
KimiBot
ai crawler
Moonshot AI
KimiBot
observe
KimiCrawler
ai scraper
Moonshot AI
KimiCrawler
observe
Kinsta
monitoring
Kinsta
kinsta-bot
allow
klaviyo
seo crawler
Klaviyo
Klaviyo
challenge
KlaviyoAIBot
ai crawler
Klaviyo
KlaviyoAIBot
observe
Knowings
other
Knowings
observe
Known Agent
ai assistant
Known Agents
Known
observe
signed
KOCMOHABT
other
KOCMOHABT
observe
KomodiaBot
other
KomodiaBot
observe
Konqueror
other
Konqueror
observe
KosmioBot
other
KosmioBot
observe
KrawlerBot
seo crawler
Krawler
KrawlerBot
challenge
KStandBot
ai assistant
URL Classification
KStandBot
observe
KunatoCrawler
ai agent
Zzazz
KunatoCrawler
observe
l9scan
other
l9scan
observe
laion-huggingface-processor
ai crawler
LAION / Hugging Face
laion-huggingface-processor
observe
LamarkBot
other
LamarkBot
observe
Landau-Media-Spider
other
Landau-Media-Spider
observe
larbin
other
larbin
observe
Laserlikebot
other
Laserlikebot
observe
lb-spider
other
lb-spider
observe
LCC
ai scraper
University of Leipzig
LCC
observe
leadbox
other
leadbox
observe
LegalMonster
security
Legal Monster ApS
LegalMonster
allow
LeikiBot
seo crawler
DoubleVerify
Leikibot
challenge
Let's
ai coding agent
Let's Encrypt
Let's
observe
Let's Encrypt
security
Let's Encrypt
allow
Level9SearchBot
search engine
Level 9
Level9SearchBot
allow
LexiBot
other
LexiBot
observe
Library Of Congress Web Archiving
ai scraper
United States Library of Congress
Library
observe