Updated daily
Crawler Catalog
2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,426 results
Newslitbot
ai assistant
Newslit
Newslitbot
observe
NewsRoom.BI
ai assistant
Marfeel
NewsRoom.BI
observe
Newzbin
other
Newzbin
observe
NextCloud
ai assistant
Nextcloud
NextCloud
observe
Nextdoorbot
other
Nextdoorbot
observe
NextGenSearchBot
other
NextGenSearchBot
observe
NICErsPRO
other
NICErsPRO
observe
niki-bot
other
niki-bot
observe
NimbleCrawler
other
NimbleCrawler
observe
Nimbostratus-Bot
other
Nimbostratus-Bot
observe
NINJA bot
other
NINJA
observe
Nitro-Optimizer-Agent
ai coding agent
NitroPack
Nitro-Optimizer-Agent
observe
Nitro-Warmup-Agent
other
Nitro-Warmup-Agent
observe
NIXStatsbot
ai coding agent
NixStats
NIXStatsbot
observe
NLUX_IAHarvester
other
NLUX_IAHarvester
observe
NoahBot
other
NoahWire
NoahBot
observe
NovaAct
ai agent
Amazon
NovaAct
observe
Novellum AI Crawl
ai crawler
Novellum
Novellum
observe
NPBot
other
NPBot
observe
NTENTbot
other
NTENTbot
observe
Nuzzel
other
Nuzzel
observe
NVBot
other
NVBot
observe
OAI-SearchBot
ai crawler
OpenAI
OAI-SearchBot
observe
OdklBot
ai assistant
VK
OdklBot
observe
officestorebot
other
officestorebot
observe
OhDear.app
ai coding agent
Oh Dear
OhDear.app
observe
oii-research
other
oii-research
observe
omgili
ai scraper
Webz.io
omgili
observe
Omgilibot
other
Omgilibot
observe
OmvionLeadLake
ai assistant
OMVION
OmvionLeadLake
observe
OnlineOrNot.com_bot
ai coding agent
OnlineOrNot
OnlineOrNot.com_bot
observe
Openbot
other
Openbot
observe
opencode
ai coding agent
Anomaly
opencode
observe
openfind
other
openfind
observe
Openfind data gatherer
other
observe
OpenGraph.io
ai assistant
Opengraph
OpenGraph.io
observe
OpenGraphCheck
other
OpenGraphCheck
observe
OpenHoseBot
other
OpenHoseBot
observe
OpenTheBoxBot
other
OpenTheBoxBot
observe
opinion-tracker
other
opinion-tracker
observe
Oracle Ultra Search
other
Oracle
observe
OrangeBot
other
OrangeBot
observe
Orbbot
other
Orbbot
observe
Orthogaffe
other
Orthogaffe
observe
osentix-crawler
ai assistant
Osentix
osentix-crawler
observe
outboundbuddy-enrichment
ai assistant
Outbound Buddy
outboundbuddy-enrichment
observe
OutclicksBot
other
OutclicksBot
observe
Overture-Maps-LinkChecker
ai coding agent
Overture Maps
Overture-Maps-LinkChecker
observe
Owler
ai assistant
Owler
Owler
observe
OWLer-W
other
OWLer-W
observe
page-preview-tool
ai assistant
SalesViewer
page-preview-tool
observe
Page2RSS
other
Page2RSS
observe
PagePeeker
ai assistant
PagePeeker
PagePeeker
observe
PageThing
other
PageThing
observe
Pandalytics
ai assistant
Domainsbot
Pandalytics
observe
PanguBot
ai scraper
Huawei
PanguBot
observe
panscient.com
ai assistant
Panscient
panscient.com
observe
Paqlebot
other
Paqle
Paqlebot
observe
parisbot
other
parisbot
observe
parsely.com
ai coding agent
Parse.ly
parsely.com
observe