Updated daily

Crawler Catalog

1,876 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

460

Allowed

1,224

Observed

192

Challenged

0

Blocked

Clear

All 1,876 Search Engine 114 AI Crawler 64 AI Assistant 268 AI Agent 10 AI Coding Agent 75 Browser Automation 3 AI Data Provider 13 AI Scraper 40 SEO Crawler 187 Social Media 73 Security 74 Monitoring 122 Archiver 21 Other 812

812 results · Other

ModatScanner other

Modat

Modular DS other

Uniqoders Technologies SL

Mollie Bot other

Mollie B.V.

MomentumFetcher other

MomentumFetcher

MonitageUptime other

MonitageUptime

Montonio Webhooks other

Montonio

MoodleBot other

MSIECrawler other

Multiviewbot other

MxToolbox other

MyEducationalCrawler other

MyEducationalCrawler

NerdByNature.Bot other

NerdByNature.Bot

NetMechanic other

NetResearchServer other

NetResearchServer

NetScope-DomainScanner other

NetScope-DomainScanner

NetSystemsResearch other

NetSystemsResearch

New York Times Newsgathering other

The New York Times

newsharecounts other

newsharecounts

NextGenSearchBot other

NextGenSearchBot

NICErsPRO other

NimbleCrawler other

NimbleCrawler

Nimbostratus-Bot other

Nimbostratus-Bot

Nitro-Warmup-Agent other

Nitro-Warmup-Agent

NLUX_IAHarvester other

NLUX_IAHarvester

NoahWire

Noibu JS Beautifier other

Noibu

Notion Automation other

Notion

officestorebot other

officestorebot

oii-research other

Omgilibot other

Omnisend

OpenGraphCheck other

OpenGraphCheck

OpenHoseBot other

OpenTheBoxBot other

OpenTheBoxBot

opinion-tracker other

opinion-tracker

OrangeBot other

← Prev 8 / 14 Next →