Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
844 results · Other
Kemvibot
other
Kemvibot
observe
Kenjin Spider
other
Kenjin
observe
Keydrop.io
other
Keydrop.io
observe
keys-so-bot
other
keys-so-bot
observe
Keyword Density
other
Keyword
observe
Knowings
other
Knowings
observe
Known Agents Browser
other
Known Agents
observe
KOCMOHABT
other
KOCMOHABT
observe
KomodiaBot
other
KomodiaBot
observe
Konqueror
other
Konqueror
observe
KosmioBot
other
KosmioBot
observe
l9scan
other
l9scan
observe
LamarkBot
other
LamarkBot
observe
Landau-Media-Spider
other
Landau-Media-Spider
observe
larbin
other
larbin
observe
Laserlikebot
other
Laserlikebot
observe
lb-spider
other
lb-spider
observe
leadbox
other
leadbox
observe
LexiBot
other
LexiBot
observe
libWeb
other
libWeb
observe
Light Crawler
other
Light
observe
Linguee Bot
other
Linguee
observe
linkapediabot
other
linkapediabot
observe
LinkArchiver
other
LinkArchiver
observe
LinkCheck by Siteimprove.com
other
observe
linkdex
other
linkdex
observe
LinkextractorPro
other
LinkextractorPro
observe
LinkisBot
other
LinkisBot
observe
linko
other
linko
observe
LinkpadBot
other
LinkpadBot
observe
LinkRing
other
LinkRing
observe
LinkScan
other
LinkScan
observe
lipperhey
other
lipperhey
observe
LivelapBot
other
LivelapBot
observe
lkxscan
other
lkxscan
observe
LNSpiderguy
other
LNSpiderguy
observe
Longtail News RSS Reader Bot
other
Longtail
observe
lssbot
other
lssbot
observe
lssrocketcrawler
other
lssrocketcrawler
observe
ltx71
other
ltx71
observe
Luminator-robots
other
Luminator-robots
observe
lwp-trivial
other
lwp-trivial
observe
MaCoCu
other
MaCoCu
observe
Magus Bot
other
Magus
observe
MainWP
other
Direct Support / Website Managed
MainWP
allow
Make Platform
other
Make
allow
MakeMerryBot
other
Make Merry
MakeMerryBot
observe
ManageWP
other
ManageWP
ManageWP
allow
mappydata
other
mappydata
observe
MastodonInstances
other
MastodonInstances
observe
Mata Hari
other
Mata
observe
MatchboxBot
other
MatchboxBot
observe
MatchorySearch
other
MatchorySearch
observe
MauiBot
other
MauiBot
observe
Mavifinds
other
Brandimi
Mavifinds
observe
MegaIndex
other
MegaIndex
observe
MelonMesa Bot
other
MelonMesa
CentComBot
observe
Meltawer
other
Meltawer
observe
Meltwater
other
Meltwater
observe
MeltwaterNews
other
MeltwaterNews
observe