Updated daily

Crawler Catalog

2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

497

Allowed

1,426

Observed

191

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

Clear

All 2,114 Search Engine 134 AI Crawler 72 AI Assistant 341 AI Agent 22 AI Coding Agent 77 Browser Automation 2 AI Data Provider 22 AI Scraper 45 SEO Crawler 193 Social Media 73 Security 82 Monitoring 122 Archiver 21 Other 908

908 results · Other

MultimodalCrawler other

MyEducationalCrawler other

MyEducationalCrawler

nameserver-checker-http-validator other

nameserver-checker-http-validator

NAVER Blog Rssbot other

NerdByNature.Bot other

NetResearchServer other

NetResearchServer

observe

NetScope-DomainScanner other

NetScope-DomainScanner

observe

NetSystemsResearch other

NetSystemsResearch

observe

New York Times Newsgathering other

The New York Times

NextGenSearchBot other

Nimbostratus-Bot other

Nitro-Warmup-Agent other

Nitro-Warmup-Agent

observe

NLUX_IAHarvester other

NoahWire

Noibu JS Beautifier other

Noibu

Noibu

allow

Notion Automation other

Notion

Omnisend

Openfind data gatherer other

opinion-tracker other

opinion-tracker

observe

Oracle Ultra Search other

Paddle Webhooks other

Paddle

← Prev 9 / 16 Next →