Updated daily
Crawler Catalog
1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
812 results · Other
KosmioBot
other
KosmioBot
observe
l9scan
other
l9scan
observe
LamarkBot
other
LamarkBot
observe
Landau-Media-Spider
other
Landau-Media-Spider
observe
larbin
other
larbin
observe
Laserlikebot
other
Laserlikebot
observe
lb-spider
other
lb-spider
observe
leadbox
other
leadbox
observe
LexiBot
other
LexiBot
observe
libWeb
other
libWeb
observe
Light
other
Light
observe
Linguee
other
Linguee
observe
linkapediabot
other
linkapediabot
observe
LinkArchiver
other
LinkArchiver
observe
linkdex
other
linkdex
observe
LinkextractorPro
other
LinkextractorPro
observe
LinkisBot
other
LinkisBot
observe
linko
other
linko
observe
LinkpadBot
other
LinkpadBot
observe
LinkRing
other
LinkRing
observe
LinkScan
other
LinkScan
observe
lipperhey
other
lipperhey
observe
LivelapBot
other
LivelapBot
observe
lkxscan
other
lkxscan
observe
LNSpiderguy
other
LNSpiderguy
observe
Longtail
other
Longtail
observe
lssbot
other
lssbot
observe
lssrocketcrawler
other
lssrocketcrawler
observe
ltx71
other
ltx71
observe
Luminator-robots
other
Luminator-robots
observe
lwp-trivial
other
lwp-trivial
observe
MaCoCu
other
MaCoCu
observe
Magus
other
Magus
observe
MainWP
other
Direct Support / Website Managed
MainWP
allow
Make Platform
other
Make
allow
MakeMerryBot
other
Make Merry
MakeMerryBot
observe
ManageWP
other
ManageWP
allow
mappydata
other
mappydata
observe
MastodonInstances
other
MastodonInstances
observe
Mata
other
Mata
observe
MatchboxBot
other
MatchboxBot
observe
MatchorySearch
other
MatchorySearch
observe
MauiBot
other
MauiBot
observe
MegaIndex
other
MegaIndex
observe
MelonMesa Bot
other
MelonMesa
observe
Meltawer
other
Meltawer
observe
Meltwater
other
Meltwater
observe
MeltwaterNews
other
MeltwaterNews
observe
memorybot
other
memorybot
observe
mention
other
mention
observe
MetadataScraper
other
MetadataScraper
observe
MetaURI
other
MetaURI
observe
Metorik
other
Metorik
allow
MIIxpc
other
MIIxpc
observe
mindUpBot
other
mindUpBot
observe
minicrawler
other
minicrawler
observe
Minoru's
other
Minoru's
observe
Mister
other
Mister
observe
mithril-crawler
other
mithril-crawler
observe
MixnodeCache
other
MixnodeCache
observe