Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,284 results
MTRobot
other
MTRobot
observe
MuckRack
ai assistant
Muck Rack
MuckRack
observe
MuckRackFeedParser
ai assistant
Muck Rack
MuckRackFeedParser
observe
Multiviewbot
other
Multiviewbot
observe
MxToolbox
other
MxToolbox
observe
MyEducationalCrawler
other
MyEducationalCrawler
observe
mytwip
other
mytwip
observe
NanoInteractive
ai assistant
Nano Interactive
NanoInteractive
observe
NapBot
other
NapBot
observe
Nava Labs ASP (Dev)
ai assistant
Nava Labs
observe
signed
NAVER Blog Rssbot
other
NAVER
observe
NaverBot
other
NaverBot
observe
Nekuda Payment Executor
ai assistant
Nekuda
nekuda-payment-executor
observe
signed
NerdByNature.Bot
other
NerdByNature.Bot
observe
NerdyBot
other
NerdyBot
observe
NetAnts
other
NetAnts
observe
NetcraftSurveyAgent
ai assistant
Netcraft
NetcraftSurveyAgent
observe
netEstate Imprint Crawler
ai crawler
netEstate
netEstate Imprint Crawler
observe
netEstate NE Crawler
ai assistant
netEstate
observe
Neticle Crawler
ai assistant
Neticle
Neticle
observe
NetMechanic
other
NetMechanic
observe
NetResearchServer
other
NetResearchServer
observe
NetScope-DomainScanner
other
NetScope-DomainScanner
observe
NetSeer crawler
ai assistant
NetSeer
NetSeer
observe
NetSystemsResearch
other
NetSystemsResearch
observe
Netvibes
ai assistant
Netvibes
Netvibes
observe
NeuwoBot
ai assistant
Neuwo
NeuwoBot
observe
New York Times Newsgathering
other
The New York Times
New
observe
NewRelicbot
ai coding agent
New Relic
NewRelicbot
observe
newsai
other
newsai
observe
NewsBlur Feed Fetcher
ai assistant
NewsBlur
observe
NewsBlur Page Fetcher
ai assistant
NewsBlur
observe
NewsBot
other
NewsBot
observe
newsharecounts
other
newsharecounts
observe
Newslitbot
ai assistant
Newslit
Newslitbot
observe
NewsRoom.BI
ai assistant
Marfeel
NewsRoom.BI
observe
Newzbin
other
Newzbin
observe
NextCloud
ai assistant
Nextcloud
NextCloud
observe
NextGenSearchBot
other
NextGenSearchBot
observe
NICErsPRO
other
NICErsPRO
observe
niki-bot
other
niki-bot
observe
NimbleCrawler
other
NimbleCrawler
observe
Nimbostratus-Bot
other
Nimbostratus-Bot
observe
NINJA bot
other
NINJA
observe
Nitro-Optimizer-Agent
ai coding agent
NitroPack
Nitro-Optimizer-Agent
observe
Nitro-Warmup-Agent
other
Nitro-Warmup-Agent
observe
NIXStatsbot
ai coding agent
NixStats
NIXStatsbot
observe
NLUX_IAHarvester
other
NLUX_IAHarvester
observe
NoahBot
other
NoahWire
NoahBot
observe
NovaAct
ai agent
Amazon
NovaAct
observe
Novellum AI Crawl
ai crawler
Novellum
Novellum
observe
NPBot
other
NPBot
observe
NTENTbot
other
NTENTbot
observe
Nuzzel
other
Nuzzel
observe
OAI-SearchBot
ai crawler
OpenAI
OAI-SearchBot
observe
OdklBot
ai assistant
VK
OdklBot
observe
officestorebot
other
officestorebot
observe
OhDear.app
ai coding agent
Oh Dear
OhDear.app
observe
oii-research
other
oii-research
observe
omgili
ai scraper
Webz.io
omgili
observe