Updated daily

Crawler Catalog

2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

497

Allowed

1,426

Observed

191

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

Clear

All 2,114 Search Engine 134 AI Crawler 72 AI Assistant 341 AI Agent 22 AI Coding Agent 77 Browser Automation 2 AI Data Provider 22 AI Scraper 45 SEO Crawler 193 Social Media 73 Security 82 Monitoring 122 Archiver 21 Other 908

1,426 results

LLMS-Txt-Scanner other

LLMS-Txt-Scanner

observe

LMArenaUnfurlBot ai assistant

LocalDomainCrawler other

LocalDomainCrawler

observe

LogicMonitor SiteMonitor ai coding agent

LogicMonitor

observe

Longtail News RSS Reader Bot other

lssrocketcrawler other

Luminator-robots other

Magnet.me-web ai assistant

Magnet

Magnet.me-web

observe

magpie-crawler ai assistant

Brandwatch

magpie-crawler

observe

MagpieRSS ai assistant

SourceForge

MailChimp ai assistant

Mailchimp

Make.com

Make Merry

ManageWP-UptimeMonitoring ai coding agent

ManageWP

ManageWP-UptimeMonitoring

observe

Mantisbot ai assistant

Mantis

Mantisbot

observe

Manus Bot ai assistant

Manus

manus-bot

observe

Manus-User browser automation

Butterfly Effect

MapTheNetBot ai assistant

MapTheNetBot

observe

Mastodon ai assistant

Mastodon

Mastodon

observe

MastodonInstances other

MathPicDatasetCrawler other

MathPicDatasetCrawler

McontextualBot ai assistant

MContextual

McontextualBot

observe

Mediapartners-Google ai assistant

Google

Mediapartners-Google

observe

MediavineMetadataParser ai assistant

Mediavine

MediavineMetadataParser

observe

Mediumbot-MetaTagFetcher ai assistant

Medium

Mediumbot-MetaTagFetcher

MelonMesa

Meta-ExternalAgent ai crawler