Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,284 results
MakeMerryBot
other
Make Merry
MakeMerryBot
observe
Mantisbot
ai assistant
Mantis
Mantisbot
observe
Manus Bot
ai assistant
Manus
manus-bot
observe
Manus-User
browser automation
Butterfly Effect
Manus-User
observe
mappydata
other
mappydata
observe
Mastodon
ai assistant
Mastodon
Mastodon
observe
MastodonInstances
other
MastodonInstances
observe
Mata Hari
other
Mata
observe
MatchboxBot
other
MatchboxBot
observe
MatchorySearch
other
MatchorySearch
observe
MauiBot
other
MauiBot
observe
Mavifinds
other
Brandimi
Mavifinds
observe
McontextualBot
ai assistant
MContextual
McontextualBot
observe
Mediapartners-Google
ai assistant
Mediapartners-Google
observe
MediavineMetadataParser
ai assistant
Mediavine
MediavineMetadataParser
observe
Mediumbot-MetaTagFetcher
ai assistant
Medium
Mediumbot-MetaTagFetcher
observe
MegaIndex
other
MegaIndex
observe
MelonMesa Bot
other
MelonMesa
CentComBot
observe
Meltawer
other
Meltawer
observe
Meltwater
other
Meltwater
observe
MeltwaterNews
other
MeltwaterNews
observe
memorybot
other
memorybot
observe
mention
other
mention
observe
Meta-ExternalAgent
ai crawler
Meta
meta-externalagent
observe
Meta-ExternalFetcher
ai assistant
Meta
meta-externalfetcher
observe
meta-webindexer
ai crawler
Meta
meta-webindexer
observe
MetadataScraper
other
MetadataScraper
observe
MetaURI
other
MetaURI
observe
micro-crawl
ai scraper
Reflection AI
micro-crawl
observe
MIIxpc
other
MIIxpc
observe
mindUpBot
other
mindUpBot
observe
minicrawler
other
minicrawler
observe
Miniflux
ai assistant
Miniflux
Miniflux
observe
Minoru's Fediverse Crawler
other
observe
Mister PiX
other
Mister
observe
MistralAI-Index
ai crawler
Mistral
MistralAI-Index
observe
MistralAI-User
ai assistant
Mistral AI
MistralAI-User
observe
MistralBot
ai scraper
Mistral
MistralBot
observe
mithril-crawler
other
mithril-crawler
observe
MixnodeCache
other
MixnodeCache
observe
MixrankBot
ai assistant
MixRank
MixrankBot
observe
mlbot
other
mlbot
observe
moatbot
other
moatbot
observe
ModatScanner
other
Modat
ModatScanner
observe
moget
other
moget
observe
mojeek
other
mojeek
observe
MomentumFetcher
other
MomentumFetcher
observe
MonitageUptime
other
MonitageUptime
observe
monitoring360
ai coding agent
360 Monitoring
monitoring360
observe
monitoring360bot
ai coding agent
360 Monitoring
monitoring360bot
observe
montastic-monitor
ai coding agent
Montastic
montastic-monitor
observe
MoodleBot
other
MoodleBot
observe
MoonshotBot
ai scraper
Moonshot AI
MoonshotBot
observe
MoonSpider
ai scraper
Moonshot AI
MoonSpider
observe
Moreover
other
Moreover
observe
Mozilla-Tabstack
ai data provider
Mozilla
Mozilla-Tabstack
observe
MS Search 4.0 Robot
other
observe
MS Search 6.0 Robot
other
observe
MSIECrawler
other
MSIECrawler
observe
msrbot
other
msrbot
observe