Updated daily

Crawler Catalog

2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

497

Allowed

1,426

Observed

191

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

All 2,114 Search Engine 134 AI Crawler 72 AI Assistant 341 AI Agent 22 AI Coding Agent 77 Browser Automation 2 AI Data Provider 22 AI Scraper 45 SEO Crawler 193 Social Media 73 Security 82 Monitoring 122 Archiver 21 Other 908

2,114 results

mwmbl search engine

Mwmbl

MxToolbox other

MyEducationalCrawler other

MyEducationalCrawler

NagetBot search engine

Naget

nameserver-checker-http-validator other

nameserver-checker-http-validator

NanoInteractive ai assistant

Nano Interactive

NanoInteractive

Nava Labs ASP (Dev) ai assistant

Nava Labs

NAVER Blog Rssbot other

Neevabot search engine

Neeva

Nekuda Payment Executor ai assistant

Nekuda

nekuda-payment-executor

NerdByNature.Bot other

NerdByNature.Bot

NestDaddyBot search engine

NestDaddy

netarkivindsamling archiver

Netarkivet

netarkivindsamling

Netcraft security

Netcraft

Netcraft SSL Server Survey security

Netcraft

NetcraftSurveyAgent ai assistant

Netcraft

NetcraftSurveyAgent

netEstate Imprint Crawler ai crawler

netEstate

netEstate Imprint Crawler

netEstate NE Crawler ai assistant

netEstate

Neticle Crawler ai assistant

Neticle

NetMechanic other

NetResearchServer other

NetResearchServer

NetScope-DomainScanner other

NetScope-DomainScanner

NetSeer crawler ai assistant

NetSeer

NetSystemsResearch other

NetSystemsResearch

Netumo monitoring

Netumo

Netvibes ai assistant

Netvibes

NETVIGIE monitoring

Netvigie

NeuwoBot ai assistant

Neuwo

New Relic monitoring

New Relic

New York Times Newsgathering other

The New York Times

NewRelicbot ai coding agent

New Relic

NewsBank archiver

NewsBank

NewsBlur social media

NewsBlur

NewsBlur Favicon Fetcher - 1692 subscribers - https:

NewsBlur Feed Fetcher ai assistant

NewsBlur

NewsBlur Page Fetcher ai assistant

NewsBlur

newsharecounts other

newsharecounts

Newslitbot ai assistant

Newslit

NewsNow search engine

NewsNowUK

newspaper seo crawler

NewsRoom.BI ai assistant

Marfeel

NextCloud ai assistant

Nextcloud

Nextdoorbot other

NextGenSearchBot other

NextGenSearchBot

Nicecrawler archiver

NiceCrawler

NICErsPRO other

Open-source

NimbleCrawler other

NimbleCrawler

Nimbostratus-Bot other

Nimbostratus-Bot

NINJA bot other

Nitro-Optimizer-Agent ai coding agent

NitroPack

Nitro-Optimizer-Agent

Nitro-Warmup-Agent other

Nitro-Warmup-Agent

← Prev 21 / 36 Next →