Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,953 results
MonitoRSS
social media
MonitoRSS
MonitoRSS
allow
Monsidobot
monitoring
Acquia
Acquia
allow
MonSpark
monitoring
MonSpark
MonSpark
allow
montastic-monitor
ai coding agent
Montastic
montastic-monitor
observe
MontasticMonitor
monitoring
Montastic
MontasticMonitor
allow
Montonio Webhooks
other
Montonio
MontonioWebhooks
allow
MoodleBot
other
MoodleBot
observe
MoonshotBot
ai scraper
Moonshot AI
MoonshotBot
observe
MoonSpider
ai scraper
Moonshot AI
MoonSpider
observe
Moreover
other
Moreover
observe
MotoMinerBot
seo crawler
Motominer
MotoMinerBot
challenge
Moz dotbot
seo crawler
Moz
DotBot
challenge
Moz rogerbot
seo crawler
Moz
rogerbot
challenge
Mozilla-Tabstack
ai data provider
Mozilla
Mozilla-Tabstack
observe
MRGbot
search engine
MRG Web Services srl
MRGbot
allow
MS Search 4.0 Robot
other
observe
MS Search 6.0 Robot
other
observe
MSIECrawler
other
MSIECrawler
observe
MSN
search engine
Microsoft
msnbot
allow
msnbot-media
search engine
Microsoft
msnbot-media
allow
msrbot
other
msrbot
observe
MTRobot
other
MTRobot
observe
Muck Rack Scraper
social media
Muck Rack
MuckRack
allow
MuckRack
ai assistant
Muck Rack
MuckRack
observe
MuckRackFeedParser
ai assistant
Muck Rack
MuckRackFeedParser
observe
Multiviewbot
other
Multiviewbot
observe
mwmbl
search engine
Mwmbl
mwmbl
allow
MxToolbox
other
MxToolbox
observe
MyEducationalCrawler
other
MyEducationalCrawler
observe
mytwip
other
mytwip
observe
NanoInteractive
ai assistant
Nano Interactive
NanoInteractive
observe
NapBot
other
NapBot
observe
Nava Labs ASP (Dev)
ai assistant
Nava Labs
observe
signed
NAVER Blog Rssbot
other
NAVER
observe
NaverBot
other
NaverBot
observe
Neevabot
search engine
Neeva
Neevabot
allow
Nekuda Payment Executor
ai assistant
Nekuda
nekuda-payment-executor
observe
signed
NerdByNature.Bot
other
NerdByNature.Bot
observe
NerdyBot
other
NerdyBot
observe
NestDaddyBot
search engine
NestDaddy
NestDaddyBot
allow
NetAnts
other
NetAnts
observe
netarkivindsamling
archiver
Netarkivet
netarkivindsamling
allow
Netcraft
security
Netcraft
Netcraft
allow
Netcraft SSL Server Survey
security
Netcraft
allow
NetcraftSurveyAgent
ai assistant
Netcraft
NetcraftSurveyAgent
observe
netEstate Imprint Crawler
ai crawler
netEstate
netEstate Imprint Crawler
observe
netEstate NE Crawler
ai assistant
netEstate
observe
Neticle Crawler
ai assistant
Neticle
Neticle
observe
NetMechanic
other
NetMechanic
observe
NetResearchServer
other
NetResearchServer
observe
NetScope-DomainScanner
other
NetScope-DomainScanner
observe
NetSeer crawler
ai assistant
NetSeer
NetSeer
observe
NetSystemsResearch
other
NetSystemsResearch
observe
Netumo
monitoring
Netumo
netumo
allow
Netvibes
ai assistant
Netvibes
Netvibes
observe
NETVIGIE
monitoring
Netvigie
NETVIGIE
allow
NeuwoBot
ai assistant
Neuwo
NeuwoBot
observe
New Relic
monitoring
New Relic
NewRelicbot
allow
New York Times Newsgathering
other
The New York Times
New
observe
NewRelicbot
ai coding agent
New Relic
NewRelicbot
observe