Updated daily
Crawler Catalog
1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
1,876 results
MistralAI-User
ai assistant
Mistral AI
MistralAI-User
observe
MistralBot
ai scraper
Mistral
MistralBot
observe
mithril-crawler
other
mithril-crawler
observe
MixnodeCache
other
MixnodeCache
observe
MixrankBot
ai assistant
MixRank
MixrankBot
observe
MJ12Bot
seo crawler
Majestic
MJ12bot
challenge
mlbot
other
mlbot
observe
moatbot
other
moatbot
observe
ModatScanner
other
Modat
ModatScanner
observe
ModernizeBot
seo crawler
ModernizeYourWebsite
ModernizeBot
challenge
Modular DS
other
Uniqoders Technologies SL
allow
moget
other
moget
observe
Mojeek
search engine
Mojeek
MojeekBot
allow
Mollie Bot
other
Mollie B.V.
allow
MomentumFetcher
other
MomentumFetcher
observe
Monitage
monitoring
Monitage
allow
MonitageUptime
other
MonitageUptime
observe
Monitis
monitoring
Monitis
monitis
allow
monitoring360
ai coding agent
360 Monitoring
monitoring360
observe
monitoring360bot
ai coding agent
360 Monitoring
monitoring360bot
observe
MonitoRSS
social media
MonitoRSS
MonitoRSS
allow
Monsidobot
monitoring
Acquia
Acquia
allow
MonSpark
monitoring
MonSpark
MonSpark
allow
MontasticMonitor
monitoring
Montastic
MontasticMonitor
allow
Montonio Webhooks
other
Montonio
allow
MoodleBot
other
MoodleBot
observe
MoonshotBot
ai scraper
Moonshot AI
MoonshotBot
observe
MoonSpider
ai scraper
Moonshot AI
MoonSpider
observe
Moreover
other
Moreover
observe
MotoMinerBot
seo crawler
Motominer
MotoMinerBot
challenge
Moz
seo crawler
Moz
Moz
challenge
Moz dotbot
seo crawler
Moz
DotBot
challenge
Moz rogerbot
seo crawler
Moz
rogerbot
challenge
Mozilla-Tabstack
ai data provider
Mozilla
Mozilla-Tabstack
observe
MRGbot
search engine
MRG Web Services srl
MRGbot
allow
MS
other
MS
observe
MSIECrawler
other
MSIECrawler
observe
MSN
search engine
Microsoft
msnbot
allow
msnbot-media
search engine
Microsoft
msnbot-media
allow
msrbot
other
msrbot
observe
MTRobot
other
MTRobot
observe
Muck Rack Scraper
social media
Muck Rack
MuckRack
allow
MuckRackFeedParser
ai assistant
Muck Rack
MuckRackFeedParser
observe
Multiviewbot
other
Multiviewbot
observe
mwmbl
search engine
Mwmbl
mwmbl
allow
MxToolbox
other
MxToolbox
observe
MyEducationalCrawler
other
MyEducationalCrawler
observe
mytwip
other
mytwip
observe
NanoInteractive
ai assistant
Nano Interactive
NanoInteractive
observe
NapBot
other
NapBot
observe
Nava Labs ASP (Dev)
ai assistant
Nava Labs
observe
signed
NAVER
other
NAVER
observe
NaverBot
other
NaverBot
observe
Neevabot
search engine
Neeva
Neevabot
allow
Nekuda Payment Executor
ai assistant
Nekuda
observe
signed
NerdByNature.Bot
other
NerdByNature.Bot
observe
NerdyBot
other
NerdyBot
observe
NestDaddyBot
search engine
NestDaddy
NestDaddyBot
allow
NetAnts
other
NetAnts
observe
netarkivindsamling
archiver
Netarkivet
netarkivindsamling
allow