Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
908 results · Other
lipperhey
other
lipperhey
observe
LivelapBot
other
LivelapBot
observe
lkxscan
other
lkxscan
observe
LLMS-Txt-Scanner
other
LLMS-Txt-Scanner
observe
LNSpiderguy
other
LNSpiderguy
observe
LocalDomainCrawler
other
LocalDomainCrawler
observe
Longtail News RSS Reader Bot
other
Longtail
observe
lssbot
other
lssbot
observe
lssrocketcrawler
other
lssrocketcrawler
observe
ltx71
other
ltx71
observe
LumiaBot
other
LumiaBot
observe
Luminator-robots
other
Luminator-robots
observe
lwp-trivial
other
lwp-trivial
observe
MaCoCu
other
MaCoCu
observe
Magus Bot
other
Magus
observe
MainWP
other
Direct Support / Website Managed
MainWP
allow
Make Platform
other
Make
allow
MakeMerryBot
other
Make Merry
MakeMerryBot
observe
ManageWP
other
ManageWP
ManageWP
allow
mappydata
other
mappydata
observe
MastodonInstances
other
MastodonInstances
observe
Mata Hari
other
Mata
observe
MatchboxBot
other
MatchboxBot
observe
MatchorySearch
other
MatchorySearch
observe
MathPicDatasetCrawler
other
MathPicDatasetCrawler
observe
Mattermost-Bot
other
Mattermost-Bot
observe
MauiBot
other
MauiBot
observe
MegaIndex
other
MegaIndex
observe
MelonMesa Bot
other
MelonMesa
CentComBot
observe
Meltawer
other
Meltawer
observe
Meltwater
other
Meltwater
observe
MeltwaterNews
other
MeltwaterNews
observe
memorybot
other
memorybot
observe
mention
other
mention
observe
MeridianBot
other
MeridianBot
observe
MetadataScraper
other
MetadataScraper
observe
MetaURI
other
MetaURI
observe
Metorik
other
Metorik
Metorik
allow
Microsoft.News.Ingestion
other
Microsoft.News.Ingestion
observe
MIIxpc
other
MIIxpc
observe
mindUpBot
other
mindUpBot
observe
minicrawler
other
minicrawler
observe
Minoru's Fediverse Crawler
other
Minoru&
observe
Mister PiX
other
Mister
observe
mithril-crawler
other
mithril-crawler
observe
MixnodeCache
other
MixnodeCache
observe
mlbot
other
mlbot
observe
moatbot
other
moatbot
observe
ModatScanner
other
Modat
ModatScanner
observe
Modular DS
other
Uniqoders Technologies SL
allow
moget
other
moget
observe
Mollie Bot
other
Mollie B.V.
allow
MomentumFetcher
other
MomentumFetcher
observe
MonitageUptime
other
MonitageUptime
observe
Montonio Webhooks
other
Montonio
MontonioWebhooks
allow
MoodleBot
other
MoodleBot
observe
Moreover
other
Moreover
observe
MS Search 4.0 Robot
other
observe
MS Search 6.0 Robot
other
observe
mShots
other
mShots
observe