Updated daily

Crawler Catalog

2,114 known bots, crawlers, and AI agents — with UA patterns, verification methods, and recommended access policies.

Not sure what "crawler" vs "scraper" vs "agent" means here? See our terminology guide →

497

Allowed

1,426

Observed

191

Challenged

The policy each agent gets before a site sets its own. Click a number to list those agents.

Clear

All 2,114 Search Engine 134 AI Crawler 72 AI Assistant 341 AI Agent 22 AI Coding Agent 77 Browser Automation 2 AI Data Provider 22 AI Scraper 45 SEO Crawler 193 Social Media 73 Security 82 Monitoring 122 Archiver 21 Other 908

1,426 results

PartnerOptimizer-bot ai assistant

PartnerOptimizer

PartnerOptimizer-bot

PavingBot other

Payhawk Invoice Fetching Agent ai assistant

Payhawk

payhawk-invoice-fetching-bot

payroll-bot ai crawler

ADP

peer39_crawler other

Peer39

peer39_crawler

Perplexity‑User ai assistant

Perplexity

Perplexity-User

PerplexityBot ai crawler

Perplexity

PerplexityBot

PetalBot ai crawler

Huawei

PhindBot ai assistant

Phind

Pingdom.com_bot ai coding agent

SolarWinds

Pingdom.com_bot

pingping.io ai coding agent

PingPing

PipericBot ai assistant

Piperic

Piqo Podsync ai assistant

Piqo

Pixalate.com other

PlagAwareBot other

Plesk screenshot bot other

PocketParser ai assistant

PodchaserParser ai assistant

Podchaser

PodchaserParser

Podimo ai assistant

Podimo

Poduptime other

Poggio-Citations ai assistant

Poggio

Poggio-Citations

Pomothy-Bot other

Poseidon Research Crawler other

PostGooBot ai assistant

PostGoo

PostitLeadDiscovery ai assistant

PostitLeadDiscovery

PoweredByBot ai assistant

Keywords Everywhere

PressEngineBot ai assistant

PressEngine

PressEngineBot

PriEcoBot other

Primalbot other

PrimeHealthBot other

PrimeHealthBot

PrivacyAwareBot other

PrivacyAwareBot

productsup.io/crawler ai assistant

Productsup

productsup.io/crawler

PrognostiXonThinMiner ai assistant

PrognostiXon

PrognostiXonThinMiner

PromptingBot other

ProPowerBot other

ProWebWalker other

PRTGCloudBot ai coding agent

Paessler

PubMatic Crawler Bot other

PulsePoint other

PulsePoint-Crawler other

PulsePoint-Crawler

pulsetic.com ai coding agent

Pulsetic

PWABuilder other

Microsoft

PWABuilderHttpAgent

PWABuilderHttpAgent ai coding agent

Microsoft

PWABuilderHttpAgent

QAtechBot ai assistant

QA.tech

qcbot ai assistant

QUIC.cloud

qodercli ai coding agent

Qoder

QualifiedBot ai crawler

Qualified.com, Inc.

Querit-SearchBot ai data provider

Querit

Querit-SearchBot

← Prev 16 / 24 Next →