Updated daily
Crawler Catalog
1,953 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.
Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →
The policy each agent gets before a site sets its own. Click a number to list those agents.
1,953 results
newsai
other
newsai
observe
NewsBank
archiver
NewsBank
NewsBank
allow
NewsBlur
social media
NewsBlur
NewsBlur Favicon Fetcher - 1692 subscribers - https:
allow
NewsBlur Feed Fetcher
ai assistant
NewsBlur
observe
NewsBlur Page Fetcher
ai assistant
NewsBlur
observe
NewsBot
other
NewsBot
observe
newsharecounts
other
newsharecounts
observe
Newslitbot
ai assistant
Newslit
Newslitbot
observe
NewsNow
search engine
NewsNowUK
NewsNow
allow
newspaper
seo crawler
newspaper
challenge
NewsRoom.BI
ai assistant
Marfeel
NewsRoom.BI
observe
Newzbin
other
Newzbin
observe
NextCloud
ai assistant
Nextcloud
NextCloud
observe
NextGenSearchBot
other
NextGenSearchBot
observe
Nicecrawler
archiver
NiceCrawler
Nicecrawler
allow
NICErsPRO
other
NICErsPRO
observe
niki-bot
other
niki-bot
observe
Nikto
security
Open-source
challenge
NimbleCrawler
other
NimbleCrawler
observe
Nimbostratus-Bot
other
Nimbostratus-Bot
observe
NINJA bot
other
NINJA
observe
Nitro-Optimizer-Agent
ai coding agent
NitroPack
Nitro-Optimizer-Agent
observe
Nitro-Warmup-Agent
other
Nitro-Warmup-Agent
observe
NitroBot
seo crawler
NitroPack Ltd
Nitro-
challenge
NIXStatsbot
ai coding agent
NixStats
NIXStatsbot
observe
NixStatsMonitoringBot
monitoring
NixStats
NixStatsMonitoringBot
allow
NLUX_IAHarvester
other
NLUX_IAHarvester
observe
Nmap Scripting Engine
security
Nmap
Nmap
allow
NoahBot
other
NoahWire
NoahBot
observe
Nodeping
monitoring
Nodeping
NodePing
allow
Noibu JS Beautifier
other
Noibu
Noibu
allow
Noorobot
seo crawler
Noor Digital Agency AB
noorobot
challenge
Nooshub
social media
Nooshub
Nooshub
allow
NostoCrawlerBot
seo crawler
Nosto
NostoCrawlerBot
challenge
Nostra
seo crawler
Nostra
nostra
challenge
Notabot
search engine
Helpfeel
Notabot
allow
Notion Automation
other
Notion
NotionAutomation
allow
NovaAct
ai agent
Amazon
NovaAct
observe
Novellum AI Crawl
ai crawler
Novellum
Novellum
observe
NPBot
other
NPBot
observe
NTENTbot
other
NTENTbot
observe
Nutch
seo crawler
Nutch
challenge
Nuzzel
other
Nuzzel
observe
nyt_scraping
seo crawler
The New York Times
nyt_scraping
challenge
OAI-AdsBot
seo crawler
OpenAI
OAI-AdsBot
challenge
OAI-SearchBot
ai crawler
OpenAI
OAI-SearchBot
observe
Observer
security
Observer
Observer
allow
Odin
security
ODIN
Odin
allow
OdklBot
ai assistant
VK
OdklBot
observe
officestorebot
other
officestorebot
observe
Offline Explorer
seo crawler
MetaProducts
Offline
challenge
Oh Dear
monitoring
Oh Dear
OhDear
allow
OhDear.app
ai coding agent
Oh Dear
OhDear.app
observe
oii-research
other
oii-research
observe
OKX-dolphin-crawler
security
OKX
OKX-dolphin-crawler
allow
omgili
ai scraper
Webz.io
omgili
observe
Omgilibot
other
Omgilibot
observe
Omnisend
other
Omnisend
Omnisend
allow
OnCrawl
seo crawler
Oncrawl
www.oncrawl.com
challenge
OnCrawl Mobile
seo crawler
Oncrawl
challenge